Oct 30 21:14:02.261068 ip-10-0-142-119 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:14:02.261079 ip-10-0-142-119 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:14:02.261085 ip-10-0-142-119 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:14:02.261307 ip-10-0-142-119 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:14:12.451050 ip-10-0-142-119 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:14:12.451067 ip-10-0-142-119 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 69f4b82df83a452b9c4709bf2089eacb -- Oct 30 21:16:14.348115 ip-10-0-142-119 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:16:14.734940 ip-10-0-142-119 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:14.734940 ip-10-0-142-119 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:16:14.734940 ip-10-0-142-119 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:14.734940 ip-10-0-142-119 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:16:14.738564 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738482 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.738564 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738564 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.738564 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738568 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738571 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738574 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738577 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738582 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738586 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738589 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738592 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738600 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738603 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738606 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738609 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738611 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738614 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738616 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738619 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738622 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738624 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738627 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.738659 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738630 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738633 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738636 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738639 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738641 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738644 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738647 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738649 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738652 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738654 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738657 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738660 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738662 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738665 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738668 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738671 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738675 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738678 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738680 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738683 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.739111 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738686 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738688 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738691 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738693 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738696 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738698 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738701 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738703 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738707 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738711 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738714 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738717 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738719 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738722 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738726 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738728 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738731 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738734 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738737 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.739666 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738739 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738742 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738744 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738747 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738750 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738752 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738755 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738758 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738760 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738762 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738765 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738767 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738778 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738781 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738784 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738786 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738789 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738791 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738794 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738796 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.740135 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738799 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738801 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738804 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738806 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738809 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738811 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738814 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738818 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738820 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738823 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738825 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738828 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738830 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738834 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738837 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738839 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738842 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738844 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738847 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.740607 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738849 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738852 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738855 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738857 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738859 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738862 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.738866 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739699 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739706 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739709 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739712 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739715 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739718 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739721 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739724 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739726 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739729 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739732 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739734 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739737 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.741780 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739740 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739742 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739745 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739748 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739751 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739753 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739756 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739759 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739761 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739764 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739766 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739769 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739771 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739774 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739777 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739781 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739784 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739786 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739789 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.742276 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739792 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739795 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739797 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739800 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739803 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739805 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739809 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739813 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739816 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739819 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739821 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739824 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739826 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739829 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739832 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739835 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739837 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739840 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739842 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.742728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739845 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739848 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739851 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739854 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739857 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739859 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739862 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739864 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739867 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739870 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739873 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739875 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739892 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739895 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739901 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739904 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739907 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739910 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739913 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739915 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.743230 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739918 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739920 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739923 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739926 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739929 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739931 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739939 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739942 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739944 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739947 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739949 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739952 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739955 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739957 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739960 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739962 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739965 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739967 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739970 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739972 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.743729 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739975 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739978 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739980 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739983 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739985 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739988 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739990 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739993 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739996 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.739998 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740001 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740003 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740006 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740008 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740018 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740529 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740534 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740538 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740543 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.744210 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740546 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740549 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740551 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740554 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740557 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740560 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740562 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740565 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740567 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740570 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740573 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740575 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740578 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740581 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740583 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740586 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740589 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740591 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740594 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740596 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.744647 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740600 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740603 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740605 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740608 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740610 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740613 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740616 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740619 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740621 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740624 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740627 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740630 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740632 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740635 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740638 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740641 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740643 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740646 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740648 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.745132 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740651 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740654 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740656 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740659 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740661 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740664 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740666 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740670 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740673 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740675 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740678 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740680 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740683 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740685 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740689 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740693 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740697 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740700 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740703 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.745578 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740705 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740708 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740710 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740713 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740715 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740718 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740721 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740724 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740726 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740728 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740731 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740733 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740736 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740739 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740742 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740744 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740747 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740749 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740751 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740754 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.746063 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740758 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740761 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740763 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740766 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740769 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740771 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740774 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740777 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740780 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740782 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740785 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740787 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740790 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740792 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740795 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740797 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740800 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740803 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740805 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.746534 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740810 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740815 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740817 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740820 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.740823 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741865 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741876 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741897 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741902 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741906 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741910 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741914 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741919 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741923 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741926 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741930 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741933 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741936 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741939 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741942 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741945 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741948 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741951 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:16:14.747028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741956 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741959 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741962 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741965 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741969 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741973 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741976 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741979 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741982 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741985 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741988 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741991 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741994 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.741997 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742002 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742005 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742008 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742011 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742014 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742017 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742022 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742025 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742028 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742031 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:16:14.747556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742036 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742040 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742044 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742047 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742050 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742053 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742056 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742058 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742062 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742065 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742068 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742071 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742075 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742079 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742082 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742085 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742088 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742091 2572 flags.go:64] FLAG: --help="false" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742094 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742098 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742101 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742104 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742108 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:16:14.748188 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742111 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742115 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742117 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742120 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742123 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742126 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742130 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742132 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742135 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742139 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742142 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742145 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742148 2572 flags.go:64] FLAG: --lock-file="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742151 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742154 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742156 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742162 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742165 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742172 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742176 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742179 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742182 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742185 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742188 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742193 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:16:14.748732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742196 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742199 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742202 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742205 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742208 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742211 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742214 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742218 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742221 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742229 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742232 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742235 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742238 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742241 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742244 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742247 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742251 2572 flags.go:64] FLAG: --port="10250" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742254 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742257 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a0085c702f29d9d9" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742260 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742263 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742266 2572 flags.go:64] FLAG: --register-node="true" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742268 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742273 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742275 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742279 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:16:14.749351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742283 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742287 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742290 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742293 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742296 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742299 2572 flags.go:64] FLAG: --runonce="false" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742302 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742305 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742308 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742310 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742313 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742316 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742321 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742324 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742327 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742330 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742333 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742336 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742339 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742342 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742345 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742350 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742354 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742357 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742361 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:16:14.749977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742364 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742367 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742370 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742373 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742376 2572 flags.go:64] FLAG: --v="2" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742381 2572 flags.go:64] FLAG: --version="false" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742385 2572 flags.go:64] FLAG: --vmodule="" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742389 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742393 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742529 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742533 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742536 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742538 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742541 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742544 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742546 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742549 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742551 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742554 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742558 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742561 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.750555 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742564 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742567 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742569 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742572 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742575 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742578 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742581 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742584 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742587 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742590 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742592 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742595 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742598 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742600 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742603 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742605 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742608 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742611 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742613 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.751102 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742616 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742619 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742621 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742624 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742627 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742629 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742632 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742634 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742637 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742639 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742642 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742646 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742648 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742651 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742653 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742656 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742658 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742661 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742663 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.751550 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742666 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742668 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742672 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742675 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742677 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742680 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742682 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742685 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742687 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742690 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742692 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742695 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742697 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742700 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742702 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742705 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742708 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742711 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742713 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.751999 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742716 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742718 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742721 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742723 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742726 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742729 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742732 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742735 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742738 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742741 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742743 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742746 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742748 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742751 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742755 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742760 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742764 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742767 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742769 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742772 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.752475 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742774 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742777 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742779 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742782 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742785 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742787 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742790 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742793 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742795 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742798 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742800 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742803 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742805 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742808 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742810 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742814 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.752961 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742818 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.742846 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742894 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742899 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742902 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742905 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742907 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742910 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742912 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742915 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742918 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742922 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742927 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742930 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742933 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742936 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.753343 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742939 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742942 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742944 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742947 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742950 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742952 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742955 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742958 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742960 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742963 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742966 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742969 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742971 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742974 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742977 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742980 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742983 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742989 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742992 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.753731 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742994 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742997 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.742999 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743002 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743005 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743007 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743009 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743012 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743015 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743019 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743022 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743025 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743027 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743030 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743032 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743035 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743037 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743040 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743043 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743045 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.754205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743048 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743050 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743053 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743055 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743059 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743062 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743064 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743067 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743069 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743072 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743077 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743079 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743082 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743084 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743087 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743089 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743092 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743094 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743097 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.754725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743099 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743102 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743106 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743109 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743111 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743114 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743116 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743119 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743121 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743124 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743126 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743129 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743132 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743134 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743137 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743139 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743142 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743144 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743148 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.755288 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743150 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743153 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743155 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743158 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743162 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743164 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743167 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743171 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743173 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743175 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743178 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743180 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743183 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743185 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.743188 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.751941 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.751958 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.751976 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.751981 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752042 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.756003 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752048 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752051 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752054 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752058 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752061 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752064 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752068 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752071 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752073 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752076 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752079 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752082 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752084 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752087 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752090 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752092 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752095 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752097 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752100 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.756512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752104 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752108 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752111 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752114 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752117 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752119 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752122 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752124 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752127 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752132 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752135 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752137 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752140 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752142 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752145 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752148 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752150 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752153 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752156 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752158 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.757252 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752161 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752163 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752166 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752169 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752171 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752174 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752177 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752180 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752182 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752185 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752188 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752190 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752193 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752202 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752205 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752209 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752213 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752216 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752218 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.757735 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752221 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752223 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752227 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752231 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752233 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752236 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752239 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752241 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752244 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752246 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752249 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752251 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752254 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752257 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752259 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752262 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752264 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752267 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752269 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.758205 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752272 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752274 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752277 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752279 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752282 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752284 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752287 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752289 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752292 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752294 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752297 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752299 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752302 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752304 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752307 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752311 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752313 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752316 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752319 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.758725 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752322 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752324 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752327 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752329 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752332 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752334 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752337 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752339 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752342 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.752370 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752514 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752519 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752522 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752525 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752527 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752530 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.759195 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752532 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752535 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752537 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752540 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752542 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752545 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752547 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752550 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752552 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752555 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752557 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752560 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752563 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752567 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752569 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752572 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752575 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752577 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752580 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752582 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.759577 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752585 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752589 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752593 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752596 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752599 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752602 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752604 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752607 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752609 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752612 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752615 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752617 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752620 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752622 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752625 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752627 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752631 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752634 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752638 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.760058 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752640 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752643 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752645 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752648 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752650 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752653 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752656 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752658 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752661 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752664 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752666 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752669 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752671 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752674 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752676 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752679 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752681 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752684 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752686 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.760500 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752689 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752691 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752694 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752696 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752699 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752701 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752704 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752706 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752709 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752711 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752714 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752717 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752719 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752722 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752724 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752726 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752729 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752731 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752733 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752736 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.761054 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752739 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752741 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752744 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752747 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752749 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752752 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752755 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752757 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752760 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752762 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752765 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752768 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752770 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752773 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752775 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752778 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752780 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752783 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752785 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.761552 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752787 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752790 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752793 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.752815 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752848 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752852 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752854 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752857 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752860 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752863 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752866 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752869 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752871 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752874 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752894 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752897 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:16:14.762032 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752900 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752903 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752906 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752909 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752912 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752914 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752917 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752919 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752922 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752924 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752927 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752929 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752932 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752934 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752937 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752939 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752942 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752944 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752947 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752950 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:16:14.762408 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752952 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752955 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752957 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752960 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752963 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752965 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752968 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752970 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752973 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752976 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752979 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752981 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752984 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752986 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752989 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752992 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752994 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.752997 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753001 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753005 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:16:14.762866 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753008 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753010 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753016 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753018 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753021 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753023 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753026 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753028 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753031 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753034 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753036 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753038 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753041 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753045 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753048 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753051 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753054 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753057 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753060 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753063 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:16:14.763348 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753065 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753068 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753070 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753073 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753075 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753078 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753080 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753083 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753086 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753089 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753091 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753094 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753096 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753100 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753104 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753106 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753109 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753111 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753116 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:16:14.763809 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753137 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753140 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753143 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753146 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753149 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753152 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753154 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753157 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753160 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753162 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753165 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753167 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753170 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753172 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:14.753175 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.753233 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.758263 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.759260 2572 server.go:1014] "Starting client certificate rotation" Oct 30 21:16:14.764374 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.759353 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:16:14.764822 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.759395 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:16:14.764822 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.761564 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:16:14.764822 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.764705 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:16:14.777406 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.777390 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:16:14.781258 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.781241 2572 log.go:25] "Validated CRI v1 image API" Oct 30 21:16:14.782268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.782242 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:16:14.785357 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.785338 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c57171df-5c7c-4356-9344-4709e4929e69:/dev/nvme0n1p3 f50dc1a9-7901-418a-8de1-36be5193434b:/dev/nvme0n1p4] Oct 30 21:16:14.785421 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.785356 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:16:14.790511 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.790394 2572 manager.go:217] Machine: {Timestamp:2026-10-30 21:16:14.789024811 +0000 UTC m=+0.348364855 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100650 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29444563aff6a14108b9dcb408ca7d SystemUUID:ec294445-63af-f6a1-4108-b9dcb408ca7d BootID:69f4b82d-f83a-452b-9c47-09bf2089eacb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:37:e3:5c:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:37:e3:5c:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:ba:06:cc:8e:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:16:14.790511 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.790513 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:16:14.790606 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.790594 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:16:14.791770 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.791749 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:16:14.791917 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.791771 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-119.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:16:14.791960 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.791920 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:16:14.791960 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.791927 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:16:14.791960 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.791956 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:16:14.792037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.791973 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:16:14.793251 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.793241 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:16:14.797742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.797731 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:16:14.797863 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.797854 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:16:14.798707 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.798698 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:16:14.798742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.798715 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:16:14.798742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.798727 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:16:14.798742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.798735 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:16:14.798855 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.798744 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:16:14.799109 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.799098 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:16:14.803350 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.803117 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:16:14.810678 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.810664 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:16:14.811574 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811562 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:16:14.811621 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811592 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:16:14.811743 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811733 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:16:14.811772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811749 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:16:14.811772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811755 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:16:14.811772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811761 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:16:14.811772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811767 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:16:14.811772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.811773 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:16:14.812454 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.812445 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:16:14.812489 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.812455 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:16:14.812489 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.812462 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:16:14.812489 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.812468 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:16:14.812489 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.812477 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:16:14.812489 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.812487 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:16:14.813850 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.813841 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:16:14.813850 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.813850 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:16:14.818035 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.818021 2572 server.go:1263] "Started kubelet" Oct 30 21:16:14.818126 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.818060 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:16:14.818229 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.818157 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:16:14.818277 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.818247 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:16:14.818720 ip-10-0-142-119 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:16:14.820804 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.820785 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:16:14.821037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.821018 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:16:14.825002 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.824980 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:16:14.828170 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.828143 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:16:14.829319 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.829302 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:16:14.829460 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.829443 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:14.830021 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830005 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:16:14.830128 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830007 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:16:14.830225 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830131 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:16:14.830225 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830216 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:16:14.830225 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830225 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:16:14.830381 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830367 2572 factory.go:153] Registering CRI-O factory Oct 30 21:16:14.830426 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830397 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 21:16:14.830469 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830448 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:16:14.830469 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830459 2572 factory.go:55] Registering systemd factory Oct 30 21:16:14.830469 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830468 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:16:14.830591 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830491 2572 factory.go:103] Registering Raw factory Oct 30 21:16:14.830591 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.830505 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 21:16:14.832088 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.832074 2572 manager.go:319] Starting recovery of all containers Oct 30 21:16:14.846434 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.846416 2572 manager.go:324] Recovery completed Oct 30 21:16:14.850354 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.850341 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:14.852069 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852053 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:14.852206 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852089 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:14.852206 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852105 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:14.852637 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852621 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:16:14.852637 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852634 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:16:14.852762 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852651 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:16:14.852762 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.852659 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:14.853355 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.853340 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:14.853434 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.853369 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:14.853434 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.853381 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:14.854949 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.854937 2572 policy_none.go:50] "Start" Oct 30 21:16:14.854987 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.854955 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:16:14.854987 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.854965 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:16:14.856986 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.856975 2572 policy_none.go:44] "Start" Oct 30 21:16:14.901491 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.901457 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:16:14.903738 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.903721 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 21:16:14.903802 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.903761 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:16:14.903802 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.903775 2572 server.go:87] "Starting device plugin registration server" Oct 30 21:16:14.904045 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.904032 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:16:14.904093 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.904046 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:16:14.904160 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.904146 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:16:14.904273 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.904254 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:16:14.904273 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.904268 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:16:14.904754 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.904735 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:16:14.904853 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.904784 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:14.957199 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.957181 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-119.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:16:14.957199 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.957181 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-119.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:16:14.958078 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.957182 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-119.us-west-1.compute.internal.18e36b3a4ffc0000 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-119.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-119.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:16:14.817779712 +0000 UTC m=+0.377119755,LastTimestamp:2026-10-30 21:16:14.817779712 +0000 UTC m=+0.377119755,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-119.us-west-1.compute.internal,}" Oct 30 21:16:14.969985 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.969823 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:16:14.971286 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.971267 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:16:14.971374 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.971289 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:16:14.971374 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:14.971313 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:16:14.971374 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:14.971352 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:16:15.004208 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.004143 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:15.004927 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.004911 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:15.005035 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.004945 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:15.005035 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.004960 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:15.005035 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.004990 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.010828 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.010745 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-119.us-west-1.compute.internal.18e36b3a520740ae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-119.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-119.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-119.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:16:14.852071598 +0000 UTC m=+0.411411653,LastTimestamp:2026-10-30 21:16:14.852071598 +0000 UTC m=+0.411411653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-119.us-west-1.compute.internal,}" Oct 30 21:16:15.011291 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.011268 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-119.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:16:15.072962 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.072933 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal"] Oct 30 21:16:15.073083 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.072991 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:15.073868 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.073848 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:15.073978 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.073898 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:15.073978 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.073913 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:15.076104 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076091 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:15.076249 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076236 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.076299 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076267 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:15.076863 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076844 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:15.076937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076847 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:15.076937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076902 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:15.076937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076916 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:15.076937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076920 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:15.076937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.076932 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:15.078222 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.078205 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:16:15.078309 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.078249 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xb96b" Oct 30 21:16:15.079112 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.079097 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.079196 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.079127 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:15.079798 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.079779 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:15.079860 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.079810 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:15.079860 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.079820 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:15.104324 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.104306 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.106902 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.106869 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.115121 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.115097 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:16:15.115221 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.115070 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-119.us-west-1.compute.internal.18e36b3a5207a1b7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-119.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-119.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-119.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:16:14.852096439 +0000 UTC m=+0.411436490,LastTimestamp:2026-10-30 21:16:14.852096439 +0000 UTC m=+0.411436490,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-119.us-west-1.compute.internal,}" Oct 30 21:16:15.115221 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.115165 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:16:15.132269 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.132245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.132367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.132272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.132367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.132306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6458e4c6fa120c378f995c71dc796f0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"d6458e4c6fa120c378f995c71dc796f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.166752 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.166722 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:15.167560 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.167543 2572 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-142-119.us-west-1.compute.internal\" not found" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.232654 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.232626 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6458e4c6fa120c378f995c71dc796f0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"d6458e4c6fa120c378f995c71dc796f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.232654 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.232657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.232835 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.232676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.232835 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.232739 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.232835 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.232752 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.232835 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.232739 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6458e4c6fa120c378f995c71dc796f0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"d6458e4c6fa120c378f995c71dc796f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.259937 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.259864 2572 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-119.us-west-1.compute.internal\" not found" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.367939 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.367905 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:15.368931 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.368910 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:15.369037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.368941 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:15.369037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.368952 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:15.369037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.368980 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.406667 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.406643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.409800 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.409781 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.425702 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.425683 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:15.425795 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.425707 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-119.us-west-1.compute.internal\": node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:15.539155 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.539079 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:15.639391 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.639360 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:15.740463 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.740434 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:15.759625 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.759598 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:16:15.759782 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:15.759758 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:16:15.759913 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.759811 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a307518d15e3f4f029c52010fdce43ee-4fb9e135dd735df7.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-119.us-west-1.compute.internal.18e36b3a520740ae\": read tcp 10.0.142.119:42198->3.137.65.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-119.us-west-1.compute.internal.18e36b3a520740ae default 5199 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-119.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-119.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-119.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:16:14 +0000 UTC,LastTimestamp:2026-10-30 21:16:15.073867134 +0000 UTC m=+0.633207176,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-119.us-west-1.compute.internal,}" Oct 30 21:16:15.841533 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.841476 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:15.942171 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:15.942143 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.043226 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.043195 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.064297 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.064282 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:16:16.065270 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.065255 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:16:16.065336 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.065288 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:16:16.065336 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.065303 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:16:16.082415 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.082379 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:11:15 +0000 UTC" deadline="2028-05-30 19:33:09.426760215 +0000 UTC" Oct 30 21:16:16.082415 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.082412 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13870h16m53.344350377s" Oct 30 21:16:16.144054 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.144030 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.244892 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.244844 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.345154 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.345115 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.414968 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.414870 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:16.446213 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.446182 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.495806 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:16.495757 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4c6d46aa36b1eea7dc39743f5eb2b7.slice/crio-94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715 WatchSource:0}: Error finding container 94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715: Status 404 returned error can't find the container with id 94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715 Oct 30 21:16:16.500184 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.500167 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:16:16.521733 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:16.521710 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6458e4c6fa120c378f995c71dc796f0.slice/crio-4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d WatchSource:0}: Error finding container 4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d: Status 404 returned error can't find the container with id 4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d Oct 30 21:16:16.547200 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.547165 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:16:16.616482 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.616453 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:16.630514 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.630486 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:16.689152 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.689078 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:16.689993 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.689976 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:16:16.690386 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.690372 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:16.748956 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.748922 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:16:16.763983 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.763963 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:16:16.800106 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.800091 2572 apiserver.go:52] "Watching apiserver" Oct 30 21:16:16.830089 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.830064 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:16:16.910629 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.910599 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:16:16.910967 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.910949 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-t6fc6","openshift-network-operator/iptables-alerter-5m86j","kube-system/konnectivity-agent-9q452","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c","openshift-dns/node-resolver-67mbf","openshift-image-registry/node-ca-5bz2b","openshift-multus/network-metrics-daemon-kzs5d","kube-system/global-pull-secret-syncer-5vztw","openshift-multus/multus-additional-cni-plugins-d2f8g","openshift-ovn-kubernetes/ovnkube-node-69lqx","openshift-cluster-node-tuning-operator/tuned-x6zm6","openshift-network-diagnostics/network-check-target-h8c58","kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal"] Oct 30 21:16:16.914780 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.914761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:16.914875 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.914851 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:16.917047 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.917032 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.919322 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.919293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:16.922079 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.922057 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.924339 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.924324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:16.926611 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.926592 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:16.929363 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.929340 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.931720 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.931701 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:16.931794 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.931776 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:16.934016 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.934000 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:16.937189 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.937168 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:16.939531 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.939484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t6fc6" Oct 30 21:16:16.941849 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.941829 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:16.941926 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:16.941884 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:16.943043 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-run\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943116 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943055 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-tmp\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943116 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943085 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.943200 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnkq\" (UniqueName: \"kubernetes.io/projected/363442c0-a599-49dd-a2d2-a0f93ac8c859-kube-api-access-xfnkq\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.943200 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943190 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-sys-fs\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.943275 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcl5\" (UniqueName: \"kubernetes.io/projected/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kube-api-access-nfcl5\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.943275 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-serviceca\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:16.943275 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943257 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-modprobe-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943494 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-host\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943494 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.943494 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqg7g\" (UniqueName: \"kubernetes.io/projected/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-kube-api-access-sqg7g\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:16.943494 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-var-lib-kubelet\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943494 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-tuned\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-cnibin\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-kubernetes\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943553 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-sys\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-agent-certs\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943626 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-konnectivity-ca\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943651 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-system-cni-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943696 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-device-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.943766 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-host\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943795 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysconfig\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-tmp-dir\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzhk\" (UniqueName: \"kubernetes.io/projected/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-kube-api-access-hhzhk\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.943997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kg9f\" (UniqueName: \"kubernetes.io/projected/12073275-f042-497b-97cc-10cf8d661a68-kube-api-access-8kg9f\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-conf\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-hosts-file\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944101 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-registration-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944142 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-systemd\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944165 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpz4\" (UniqueName: \"kubernetes.io/projected/b29ee7c6-efd8-4404-8a4d-de677037375e-kube-api-access-twpz4\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:16.944268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-lib-modules\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:16.944751 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-socket-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.944751 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.944327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:16.972657 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.972635 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:16:16.972657 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.972640 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:16.972823 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.972679 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fp6d9\"" Oct 30 21:16:16.974402 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.974378 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:16:16.974555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.974533 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8g8v4\"" Oct 30 21:16:16.974657 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.974619 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:16:16.976286 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.976240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"d6458e4c6fa120c378f995c71dc796f0","Type":"ContainerStarted","Data":"4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d"} Oct 30 21:16:16.977018 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.976984 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:16:16.977301 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.977281 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:16:16.977402 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.977382 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:16:16.977402 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.977388 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrcct\"" Oct 30 21:16:16.977533 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.977514 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"4a4c6d46aa36b1eea7dc39743f5eb2b7","Type":"ContainerStarted","Data":"94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715"} Oct 30 21:16:16.979280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.979265 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ztz2s\"" Oct 30 21:16:16.979344 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.979272 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:16:16.979388 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.979374 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:16:16.981838 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.981820 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:16:16.981898 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.981826 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6llv\"" Oct 30 21:16:16.981898 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.981854 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:16:16.981986 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.981904 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:16:16.984565 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.984548 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kknvl\"" Oct 30 21:16:16.984646 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.984553 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:16:16.984705 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.984666 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:16:16.984764 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:16.984726 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:16:17.032276 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.032248 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:16:17.044736 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-systemd-units\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.044810 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-k8s-cni-cncf-io\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.044858 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-log-socket\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.044858 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-kubernetes\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.044985 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044869 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-sys\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.044985 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044885 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-konnectivity-ca\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:17.044985 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-device-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.044985 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-slash\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.044985 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044940 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-kubernetes\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044962 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.044994 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-sys\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-device-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-tmp-dir\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzhk\" (UniqueName: \"kubernetes.io/projected/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-kube-api-access-hhzhk\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-env-overrides\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.045209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4229762d-c3ea-43fd-851d-506bba6c76db-ovn-node-metrics-cert\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.045548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwdj2\" (UniqueName: \"kubernetes.io/projected/4229762d-c3ea-43fd-851d-506bba6c76db-kube-api-access-zwdj2\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.045548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-conf-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.045548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045282 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-conf\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.045548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-hosts-file\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.045548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045425 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-conf\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.045548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045443 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-hosts-file\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.045815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045788 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-multus\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.045873 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045823 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-multus-daemon-config\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.045873 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmz2\" (UniqueName: \"kubernetes.io/projected/2d658122-115c-4a45-965d-5d75e43ff1be-kube-api-access-csmz2\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.046001 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-kubelet-config\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.046001 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-etc-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.046001 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-system-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.046001 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045972 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-tmp-dir\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.045998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-kubelet\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046030 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-etc-kubernetes\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d658122-115c-4a45-965d-5d75e43ff1be-iptables-alerter-script\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-run\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046145 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-konnectivity-ca\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.046209 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-sys-fs\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcl5\" (UniqueName: \"kubernetes.io/projected/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kube-api-access-nfcl5\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046239 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046253 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-sys-fs\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046278 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-dbus\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046314 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-run\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046325 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-kubelet\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-bin\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046386 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-modprobe-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-host\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046456 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046441 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-modprobe-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046459 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-host\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046485 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-cnibin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-hostroot\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-var-lib-kubelet\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-tuned\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-cnibin\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046585 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046615 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-cnibin\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rfbd\" (UniqueName: \"kubernetes.io/projected/506349da-6a0c-482a-a953-3eaa76e552b6-kube-api-access-5rfbd\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046641 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-var-lib-kubelet\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046661 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-var-lib-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-agent-certs\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046741 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-system-cni-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.046834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-host\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046807 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-system-cni-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046797 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-host\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-config\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046808 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-script-lib\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysconfig\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.046984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047010 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-systemd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-netd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047041 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysconfig\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-bin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047088 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kg9f\" (UniqueName: \"kubernetes.io/projected/12073275-f042-497b-97cc-10cf8d661a68-kube-api-access-8kg9f\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-registration-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.047601 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-systemd\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047220 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-systemd\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpz4\" (UniqueName: \"kubernetes.io/projected/b29ee7c6-efd8-4404-8a4d-de677037375e-kube-api-access-twpz4\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d658122-115c-4a45-965d-5d75e43ff1be-host-slash\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047345 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-registration-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-lib-modules\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-socket-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047466 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-node-log\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047494 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047505 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-netns\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-multus-certs\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047554 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-socket-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047584 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-lib-modules\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-tmp\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.048372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnkq\" (UniqueName: \"kubernetes.io/projected/363442c0-a599-49dd-a2d2-a0f93ac8c859-kube-api-access-xfnkq\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047692 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-serviceca\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047807 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg7g\" (UniqueName: \"kubernetes.io/projected/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-kube-api-access-sqg7g\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047833 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-netns\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.047839 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.047903 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:17.547884954 +0000 UTC m=+3.107225010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.047992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-ovn\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.048026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-cni-binary-copy\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.048063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-socket-dir-parent\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.048088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.048897 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.048279 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-serviceca\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.050411 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.050392 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-tuned\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.050557 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.050532 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-tmp\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.050779 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.050761 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-agent-certs\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:17.074312 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.074283 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:16:17.074423 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.074332 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x42zp" Oct 30 21:16:17.082368 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082333 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmnhg\"" Oct 30 21:16:17.082368 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082347 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:16:17.082368 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082359 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:16:17.082368 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082338 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:16:17.082633 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082367 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:16:17.082633 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082395 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:16:17.082633 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082540 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082838 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082846 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-msgtv\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082841 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082848 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082872 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082891 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:16:17.082945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082948 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ks46b\"" Oct 30 21:16:17.083204 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.082972 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:16:17.085654 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.085635 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.087934 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.087893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.105900 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.105871 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:17.106014 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.105890 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:17.106014 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.105919 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:17.106014 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.105983 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:17.605964477 +0000 UTC m=+3.165304550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:17.107103 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.106916 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzhk\" (UniqueName: \"kubernetes.io/projected/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-kube-api-access-hhzhk\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.107922 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.107881 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcl5\" (UniqueName: \"kubernetes.io/projected/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kube-api-access-nfcl5\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.108061 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.108044 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kg9f\" (UniqueName: \"kubernetes.io/projected/12073275-f042-497b-97cc-10cf8d661a68-kube-api-access-8kg9f\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:17.108405 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.108387 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg7g\" (UniqueName: \"kubernetes.io/projected/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-kube-api-access-sqg7g\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.109108 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.109085 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpz4\" (UniqueName: \"kubernetes.io/projected/b29ee7c6-efd8-4404-8a4d-de677037375e-kube-api-access-twpz4\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.109108 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.109106 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnkq\" (UniqueName: \"kubernetes.io/projected/363442c0-a599-49dd-a2d2-a0f93ac8c859-kube-api-access-xfnkq\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.149293 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d658122-115c-4a45-965d-5d75e43ff1be-host-slash\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.149293 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-node-log\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.149564 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.149635 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-netns\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.149691 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149651 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d658122-115c-4a45-965d-5d75e43ff1be-host-slash\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.149742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-multus-certs\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.149742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149699 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-node-log\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.149742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149700 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.149742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149723 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-netns\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.149742 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-netns\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-ovn\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149788 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-netns\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149825 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-ovn\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-cni-binary-copy\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149869 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-socket-dir-parent\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149914 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150003 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.149952 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-systemd-units\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-k8s-cni-cncf-io\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150048 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-systemd-units\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-log-socket\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150108 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-slash\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150168 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150170 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-socket-dir-parent\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-env-overrides\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150216 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-k8s-cni-cncf-io\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4229762d-c3ea-43fd-851d-506bba6c76db-ovn-node-metrics-cert\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150254 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-slash\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdj2\" (UniqueName: \"kubernetes.io/projected/4229762d-c3ea-43fd-851d-506bba6c76db-kube-api-access-zwdj2\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150290 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-log-socket\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-conf-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-multus\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-multus-daemon-config\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150394 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmz2\" (UniqueName: \"kubernetes.io/projected/2d658122-115c-4a45-965d-5d75e43ff1be-kube-api-access-csmz2\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-kubelet-config\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-etc-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-system-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-kubelet\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150551 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-cni-binary-copy\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-etc-kubernetes\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d658122-115c-4a45-965d-5d75e43ff1be-iptables-alerter-script\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-dbus\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-kubelet\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150736 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150734 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-bin\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150796 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-cnibin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150798 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-bin\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-hostroot\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.150910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150849 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-env-overrides\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150875 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150920 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfbd\" (UniqueName: \"kubernetes.io/projected/506349da-6a0c-482a-a953-3eaa76e552b6-kube-api-access-5rfbd\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.150954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-var-lib-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.150990 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151007 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-config\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.151061 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:17.651041952 +0000 UTC m=+3.210382000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-script-lib\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151149 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-systemd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151187 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-netd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-bin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151311 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-bin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-cnibin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151418 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-hostroot\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151491 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-netd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151499 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-multus-certs\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.151709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-var-lib-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151588 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-config\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151643 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.151707 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-etc-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152043 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-script-lib\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152097 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-systemd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152167 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-conf-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-multus\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152244 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-etc-kubernetes\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152286 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-kubelet-config\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152313 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-system-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152325 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-multus-daemon-config\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152345 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-kubelet\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152398 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-kubelet\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.152495 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152414 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-dbus\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.153128 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.152829 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d658122-115c-4a45-965d-5d75e43ff1be-iptables-alerter-script\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.155665 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.155644 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4229762d-c3ea-43fd-851d-506bba6c76db-ovn-node-metrics-cert\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.213306 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.213233 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfbd\" (UniqueName: \"kubernetes.io/projected/506349da-6a0c-482a-a953-3eaa76e552b6-kube-api-access-5rfbd\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.213306 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.213249 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdj2\" (UniqueName: \"kubernetes.io/projected/4229762d-c3ea-43fd-851d-506bba6c76db-kube-api-access-zwdj2\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.213306 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.213233 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmz2\" (UniqueName: \"kubernetes.io/projected/2d658122-115c-4a45-965d-5d75e43ff1be-kube-api-access-csmz2\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.228685 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.228663 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:16:17.235213 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.235197 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:17.239327 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.239305 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29ee7c6_efd8_4404_8a4d_de677037375e.slice/crio-fdd964ddf22e41ece806f274cd93c4650ee088fd4a28cef6c2428e9e1a31d43a WatchSource:0}: Error finding container fdd964ddf22e41ece806f274cd93c4650ee088fd4a28cef6c2428e9e1a31d43a: Status 404 returned error can't find the container with id fdd964ddf22e41ece806f274cd93c4650ee088fd4a28cef6c2428e9e1a31d43a Oct 30 21:16:17.244512 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.244489 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8f39f3_c5f2_4cde_8a5d_05cd13829521.slice/crio-2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1 WatchSource:0}: Error finding container 2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1: Status 404 returned error can't find the container with id 2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1 Oct 30 21:16:17.249866 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.249849 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:16:17.250974 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.250929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-67mbf" Oct 30 21:16:17.255780 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.255762 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:16:17.261068 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.261044 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59caf8ab_703a_4d27_bf2c_faa75cdf4c36.slice/crio-58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c WatchSource:0}: Error finding container 58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c: Status 404 returned error can't find the container with id 58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c Oct 30 21:16:17.265430 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.265406 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0fea2c_ba27_4a1c_a48d_c070d9577b1b.slice/crio-d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa WatchSource:0}: Error finding container d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa: Status 404 returned error can't find the container with id d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa Oct 30 21:16:17.266705 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.266693 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a1b8a3_1ae3_4724_ba2e_7260a79c6812.slice/crio-e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f WatchSource:0}: Error finding container e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f: Status 404 returned error can't find the container with id e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f Oct 30 21:16:17.275237 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.275216 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:16:17.279152 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.279139 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:16:17.281686 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.281661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t6fc6" Oct 30 21:16:17.286010 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.285984 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d658122_115c_4a45_965d_5d75e43ff1be.slice/crio-355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4 WatchSource:0}: Error finding container 355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4: Status 404 returned error can't find the container with id 355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4 Oct 30 21:16:17.291643 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.291617 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363442c0_a599_49dd_a2d2_a0f93ac8c859.slice/crio-9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3 WatchSource:0}: Error finding container 9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3: Status 404 returned error can't find the container with id 9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3 Oct 30 21:16:17.292220 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.292199 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506349da_6a0c_482a_a953_3eaa76e552b6.slice/crio-2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11 WatchSource:0}: Error finding container 2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11: Status 404 returned error can't find the container with id 2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11 Oct 30 21:16:17.297612 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.297595 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:17.307451 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:16:17.307431 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4229762d_c3ea_43fd_851d_506bba6c76db.slice/crio-18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3 WatchSource:0}: Error finding container 18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3: Status 404 returned error can't find the container with id 18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3 Oct 30 21:16:17.554970 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.554872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:17.555152 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.555122 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:17.555226 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.555214 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:18.555192598 +0000 UTC m=+4.114532637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:17.655902 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.655854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:17.655902 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.655902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:17.656150 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.656066 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:17.656150 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.656081 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:17.656150 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.656090 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:17.656315 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.656155 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:18.656132591 +0000 UTC m=+4.215472637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:17.656315 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.656237 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:17.656315 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:17.656276 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:18.656264208 +0000 UTC m=+4.215604243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:17.980768 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.980725 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerStarted","Data":"9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3"} Oct 30 21:16:17.982485 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.982452 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9q452" event={"ID":"0b8f39f3-c5f2-4cde-8a5d-05cd13829521","Type":"ContainerStarted","Data":"2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1"} Oct 30 21:16:17.983917 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.983871 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" event={"ID":"b29ee7c6-efd8-4404-8a4d-de677037375e","Type":"ContainerStarted","Data":"fdd964ddf22e41ece806f274cd93c4650ee088fd4a28cef6c2428e9e1a31d43a"} Oct 30 21:16:17.986229 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.986185 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5m86j" event={"ID":"2d658122-115c-4a45-965d-5d75e43ff1be","Type":"ContainerStarted","Data":"355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4"} Oct 30 21:16:17.987582 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.987560 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bz2b" event={"ID":"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812","Type":"ContainerStarted","Data":"e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f"} Oct 30 21:16:17.989180 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.989149 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-67mbf" event={"ID":"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b","Type":"ContainerStarted","Data":"d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa"} Oct 30 21:16:17.990580 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.990550 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" event={"ID":"59caf8ab-703a-4d27-bf2c-faa75cdf4c36","Type":"ContainerStarted","Data":"58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c"} Oct 30 21:16:17.992243 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.992206 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3"} Oct 30 21:16:17.993407 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:17.993384 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t6fc6" event={"ID":"506349da-6a0c-482a-a953-3eaa76e552b6","Type":"ContainerStarted","Data":"2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11"} Oct 30 21:16:18.079163 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.079113 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:11:17 +0000 UTC" deadline="2028-05-15 10:28:59.622587273 +0000 UTC" Oct 30 21:16:18.079163 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.079155 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13501h12m41.543437445s" Oct 30 21:16:18.564917 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.564818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:18.565136 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.564989 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:18.565136 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.565055 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:20.56503598 +0000 UTC m=+6.124376027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:18.666168 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.666033 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:18.666359 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.666186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:18.666359 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.666306 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:18.666486 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.666370 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:20.66635081 +0000 UTC m=+6.225690857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:18.666486 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.666464 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:18.666486 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.666480 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:18.666634 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.666492 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:18.666634 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.666530 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:20.666517676 +0000 UTC m=+6.225857722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:18.973123 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.973089 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:18.973309 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.973249 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:18.973382 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.973342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:18.973436 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.973421 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:18.975355 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:18.975327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:18.975488 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:18.975455 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:19.080034 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:19.079971 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:11:17 +0000 UTC" deadline="2028-08-11 17:10:02.791288794 +0000 UTC" Oct 30 21:16:19.080034 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:19.080028 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15619h53m43.711263761s" Oct 30 21:16:20.585932 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:20.585354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:20.585932 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.585770 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:20.585932 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.585841 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:24.585821155 +0000 UTC m=+10.145161215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:20.686197 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:20.686152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:20.686405 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:20.686221 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:20.686405 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.686384 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:20.686405 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.686405 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:20.686575 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.686420 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:20.686575 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.686488 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:24.686467805 +0000 UTC m=+10.245807843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:20.686575 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.686555 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:20.686739 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.686595 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:24.686583305 +0000 UTC m=+10.245923337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:20.974768 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:20.974731 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:20.974999 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.974884 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:20.974999 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:20.974948 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:20.975129 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.975056 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:20.975129 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:20.975102 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:20.975229 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:20.975165 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:21.040707 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:21.040663 2572 generic.go:356] "Generic (PLEG): container finished" podID="4a4c6d46aa36b1eea7dc39743f5eb2b7" containerID="2b45e00b3eb9e91d8cc22770d9e2e9e506523e14511e68928d88bf2dd0a9a0e1" exitCode=0 Oct 30 21:16:21.040892 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:21.040781 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"4a4c6d46aa36b1eea7dc39743f5eb2b7","Type":"ContainerDied","Data":"2b45e00b3eb9e91d8cc22770d9e2e9e506523e14511e68928d88bf2dd0a9a0e1"} Oct 30 21:16:21.049665 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:21.049618 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"d6458e4c6fa120c378f995c71dc796f0","Type":"ContainerStarted","Data":"3d382d92caded67191c88364acab0a437f6e6ae38784b585e28e44a123b35898"} Oct 30 21:16:22.060819 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.060775 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"4a4c6d46aa36b1eea7dc39743f5eb2b7","Type":"ContainerStarted","Data":"72da923226da103f5c9c1effee3b81fe45f8992c3f12832e1d83fda3b3989f85"} Oct 30 21:16:22.064835 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.064803 2572 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="7f9752885c9f7c7b9c70c0bfe900708f68829dc75623886041656213d87b9a23" exitCode=0 Oct 30 21:16:22.066126 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.066099 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerDied","Data":"7f9752885c9f7c7b9c70c0bfe900708f68829dc75623886041656213d87b9a23"} Oct 30 21:16:22.183052 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.182907 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" podStartSLOduration=5.182887482 podStartE2EDuration="5.182887482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:21.282760758 +0000 UTC m=+6.842100815" watchObservedRunningTime="2026-10-30 21:16:22.182887482 +0000 UTC m=+7.742227539" Oct 30 21:16:22.183253 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.183133 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" podStartSLOduration=5.183121632 podStartE2EDuration="5.183121632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:22.182095898 +0000 UTC m=+7.741435954" watchObservedRunningTime="2026-10-30 21:16:22.183121632 +0000 UTC m=+7.742461689" Oct 30 21:16:22.974741 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.974702 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:22.974931 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:22.974854 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:22.975024 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.974961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:22.975080 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:22.975046 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:22.975138 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:22.975113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:22.975218 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:22.975197 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:23.071862 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:23.071828 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9q452" event={"ID":"0b8f39f3-c5f2-4cde-8a5d-05cd13829521","Type":"ContainerStarted","Data":"e3b0fba625907eee531ba274f3e49286501e836ef7154793b29f1efe1320f878"} Oct 30 21:16:24.077213 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.077120 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5bz2b" event={"ID":"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812","Type":"ContainerStarted","Data":"aef74112839a510776b0ce9684982db2027740f5fc1d88f6333c50a1a668e08c"} Oct 30 21:16:24.080002 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.079972 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" event={"ID":"59caf8ab-703a-4d27-bf2c-faa75cdf4c36","Type":"ContainerStarted","Data":"6302e67ade9c8b2bb0ce520f484fbb750f856a64324ca3116f0bcd6772ceab3a"} Oct 30 21:16:24.188750 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.188656 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9q452" podStartSLOduration=4.16997565 podStartE2EDuration="9.188634819s" totalImagesPullingTime="5.018659169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:23.182760987 +0000 UTC m=+8.742101041" watchObservedRunningTime="2026-10-30 21:16:24.188634819 +0000 UTC m=+9.747974875" Oct 30 21:16:24.621111 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.621077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:24.621294 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.621216 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:24.621294 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.621280 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:32.621261489 +0000 UTC m=+18.180601533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:24.722438 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.722402 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:24.722604 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.722455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:24.722678 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.722623 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:24.722678 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.722639 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:24.722678 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.722652 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:24.722821 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.722707 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:32.722689864 +0000 UTC m=+18.282029913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:24.722821 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.722771 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:24.722821 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.722802 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:32.722792212 +0000 UTC m=+18.282132246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:24.973952 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.973919 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:24.974129 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.974017 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:24.974129 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:24.974073 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:24.974865 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.974830 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:24.974987 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.974933 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:24.975113 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:24.975024 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:26.013027 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.012907 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:16:26.094194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.094159 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" event={"ID":"59caf8ab-703a-4d27-bf2c-faa75cdf4c36","Type":"ContainerStarted","Data":"25f2812f1af7b39a09eeba86c1c0426b21556ddfbf702c90d8aa5946357b99ca"} Oct 30 21:16:26.096459 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.096427 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t6fc6" event={"ID":"506349da-6a0c-482a-a953-3eaa76e552b6","Type":"ContainerStarted","Data":"653fca5845bd5755f7e8cba8d71a2a10dc23af7d03c6d94ea34c92ab9e70c9bc"} Oct 30 21:16:26.212910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.210576 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t6fc6" podStartSLOduration=2.50819636 podStartE2EDuration="11.210562333s" totalImagesPullingTime="8.702365973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:26.210128229 +0000 UTC m=+11.769468282" watchObservedRunningTime="2026-10-30 21:16:26.210562333 +0000 UTC m=+11.769902385" Oct 30 21:16:26.212910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.210810 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5bz2b" podStartSLOduration=5.238162268 podStartE2EDuration="11.210803766s" totalImagesPullingTime="5.972641498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:24.189112961 +0000 UTC m=+9.748453012" watchObservedRunningTime="2026-10-30 21:16:26.210803766 +0000 UTC m=+11.770143815" Oct 30 21:16:26.914236 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.914133 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:16:26.012924384Z","UUID":"53539715-47ad-4503-b874-3bd3e2708cc7","Handler":null,"Name":"","Endpoint":""} Oct 30 21:16:26.917429 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.917397 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:16:26.917429 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.917432 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:16:26.971855 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.971823 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:26.971855 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.971856 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:26.972100 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:26.971976 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:26.972100 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:26.972037 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:26.972191 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:26.972149 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:26.972269 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:26.972247 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:27.101590 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:27.100941 2572 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="db6c816284df1924a75024b182ffd66f64b7ec08b3d3fe88142ee2b3b4f8e2ef" exitCode=0 Oct 30 21:16:27.101590 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:27.101004 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerDied","Data":"db6c816284df1924a75024b182ffd66f64b7ec08b3d3fe88142ee2b3b4f8e2ef"} Oct 30 21:16:27.105158 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:27.105131 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-67mbf" event={"ID":"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b","Type":"ContainerStarted","Data":"3229852f6853f260cc9109089d42a3d323557f1d463a8100eaaa1ebbcc3120ca"} Oct 30 21:16:27.333914 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:27.333474 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-67mbf" podStartSLOduration=3.288901975 podStartE2EDuration="12.333459852s" totalImagesPullingTime="9.044557877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:27.333072129 +0000 UTC m=+12.892412194" watchObservedRunningTime="2026-10-30 21:16:27.333459852 +0000 UTC m=+12.892799905" Oct 30 21:16:27.438162 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:27.438131 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:27.438659 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:27.438638 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:28.110539 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.110505 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5m86j" event={"ID":"2d658122-115c-4a45-965d-5d75e43ff1be","Type":"ContainerStarted","Data":"8e88a7563958dfcbf586e7112e9fee744a509ab143985a3d101d5ceef4db5251"} Oct 30 21:16:28.113753 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.113722 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" event={"ID":"59caf8ab-703a-4d27-bf2c-faa75cdf4c36","Type":"ContainerStarted","Data":"5d2e8d0e28150fda5b914cb4496c58d4d56310b77bfaaa738b78d716ab9550e6"} Oct 30 21:16:28.114217 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.114191 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:28.114741 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.114715 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:16:28.223721 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.223672 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5m86j" podStartSLOduration=4.521136841 podStartE2EDuration="13.223656506s" totalImagesPullingTime="8.702519665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:28.223217921 +0000 UTC m=+13.782557976" watchObservedRunningTime="2026-10-30 21:16:28.223656506 +0000 UTC m=+13.782996559" Oct 30 21:16:28.337154 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.336853 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" podStartSLOduration=3.493309718 podStartE2EDuration="13.336836004s" totalImagesPullingTime="9.843526286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:28.336707617 +0000 UTC m=+13.896047675" watchObservedRunningTime="2026-10-30 21:16:28.336836004 +0000 UTC m=+13.896176057" Oct 30 21:16:28.972586 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.972557 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:28.972586 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.972578 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:28.972798 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:28.972655 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:28.972798 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:28.972754 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:28.973154 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:28.973132 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:28.973227 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:28.973189 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:29.117826 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:29.117789 2572 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="f2e1606b7b4cfaefdf2af00e08b661ea2c170528846df5354a8e7324a9be04e7" exitCode=0 Oct 30 21:16:29.118353 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:29.117891 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerDied","Data":"f2e1606b7b4cfaefdf2af00e08b661ea2c170528846df5354a8e7324a9be04e7"} Oct 30 21:16:29.120321 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:29.120290 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" event={"ID":"b29ee7c6-efd8-4404-8a4d-de677037375e","Type":"ContainerStarted","Data":"b7be04a1ba1b794a2a0dcdf429247299128913c09b1062390871b2b2c6be9752"} Oct 30 21:16:29.348585 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:29.348485 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" podStartSLOduration=3.372174081 podStartE2EDuration="14.348463711s" totalImagesPullingTime="10.97628963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:29.348156185 +0000 UTC m=+14.907496236" watchObservedRunningTime="2026-10-30 21:16:29.348463711 +0000 UTC m=+14.907803769" Oct 30 21:16:30.972280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:30.972242 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:30.972280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:30.972262 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:30.973136 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:30.972378 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:30.973136 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:30.972376 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:30.973136 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:30.972510 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:30.973136 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:30.972581 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:31.127739 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:31.127670 2572 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="3fea154c923779c3eac4a2e3a0a05f1b04be65fbc806c3cebef0211632161d36" exitCode=0 Oct 30 21:16:31.127739 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:31.127712 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerDied","Data":"3fea154c923779c3eac4a2e3a0a05f1b04be65fbc806c3cebef0211632161d36"} Oct 30 21:16:32.681786 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:32.681608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:32.682149 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.681760 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:32.682149 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.681929 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:48.681906887 +0000 UTC m=+34.241246939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:32.782609 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:32.782523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:32.782735 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:32.782598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:32.782735 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.782688 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:32.782833 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.782761 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:16:48.782741974 +0000 UTC m=+34.342082021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:32.782833 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.782803 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:32.782833 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.782827 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:32.782985 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.782841 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:32.782985 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.782914 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:16:48.782874444 +0000 UTC m=+34.342214493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:32.971779 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:32.971720 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:32.971779 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:32.971737 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:32.971950 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:32.971920 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:32.971950 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.971926 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:32.972072 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.972047 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:32.972166 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:32.972134 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:33.135357 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135333 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:16:33.135711 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135690 2572 generic.go:356] "Generic (PLEG): container finished" podID="4229762d-c3ea-43fd-851d-506bba6c76db" containerID="e8ae4035fa0ad1e575f66d36ba1f0049b00f7f0e627356bfd394edec9d1dfd5b" exitCode=1 Oct 30 21:16:33.135778 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135733 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"2f71055d66068be677df165b065a0e1c1cb723d67a04827af5af610637fb0aa6"} Oct 30 21:16:33.135818 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135792 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"b4bc5a2ffc0e1067d6faad6c23a07b161a155dbbedea5dec25b51b1c77c362dd"} Oct 30 21:16:33.135862 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135822 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"c1670963e9fbfdf581b962377eeb36e09b915d2194d0f189202c43a238541144"} Oct 30 21:16:33.135862 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135851 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"c8fba3e178321e611d3d10250359edd7bf08e0c737c657508d4fc93cecf4374e"} Oct 30 21:16:33.136030 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135898 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerDied","Data":"e8ae4035fa0ad1e575f66d36ba1f0049b00f7f0e627356bfd394edec9d1dfd5b"} Oct 30 21:16:33.136030 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:33.135928 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"7c5800a212a3f8bb70b44a3ef42e2f31218817cbb443c1e0e9b5e3ab41f39ac2"} Oct 30 21:16:34.140952 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:34.140759 2572 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="1248e4f320cd33795eccdb1425f6193de2125cc316d38ef1bfc04b76719b4438" exitCode=0 Oct 30 21:16:34.140952 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:34.140837 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerDied","Data":"1248e4f320cd33795eccdb1425f6193de2125cc316d38ef1bfc04b76719b4438"} Oct 30 21:16:34.976957 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:34.976928 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:34.977098 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:34.976969 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:34.977098 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:34.977053 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:34.977194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:34.977100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:34.977194 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:34.977163 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:34.977251 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:34.977227 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:35.145689 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:35.145660 2572 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="6d3f1dc8ec9b1c8549ad9a786b42338368bcd184b44e48b05eb70838d1407946" exitCode=0 Oct 30 21:16:35.146148 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:35.145716 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerDied","Data":"6d3f1dc8ec9b1c8549ad9a786b42338368bcd184b44e48b05eb70838d1407946"} Oct 30 21:16:36.150853 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.150815 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" event={"ID":"363442c0-a599-49dd-a2d2-a0f93ac8c859","Type":"ContainerStarted","Data":"f928df724aecf85a490d5a4442218217d4fc26d68c24812981ceb81699b423c7"} Oct 30 21:16:36.153636 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.153616 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:16:36.153949 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.153930 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"de4155614b824e4044e6be982ed803ed3c88593ca5dbfd0767bf7f22f9b0f494"} Oct 30 21:16:36.267336 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.267283 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d2f8g" podStartSLOduration=8.376776556 podStartE2EDuration="21.26726715s" totalImagesPullingTime="12.890490594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:36.267045282 +0000 UTC m=+21.826385334" watchObservedRunningTime="2026-10-30 21:16:36.26726715 +0000 UTC m=+21.826607202" Oct 30 21:16:36.972166 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.972126 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:36.972363 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.972173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:36.972363 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:36.972237 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:36.972363 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:36.972256 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:36.972363 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:36.972335 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:36.972556 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:36.972388 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:38.161530 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.161499 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:16:38.162333 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.161925 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"2ca9d4c46057b6c6dfe7a01c7dabe7790ea30c40aa7b8be69241681cb4f64817"} Oct 30 21:16:38.162333 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.162245 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:38.162435 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.162340 2572 scope.go:122] "RemoveContainer" containerID="e8ae4035fa0ad1e575f66d36ba1f0049b00f7f0e627356bfd394edec9d1dfd5b" Oct 30 21:16:38.179541 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.179521 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:38.971814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.971777 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:38.971814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.971798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:38.972055 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:38.971905 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:38.972055 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:38.971920 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:38.972055 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:38.971973 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:38.972179 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:38.972099 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:39.168005 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:39.167969 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:16:39.168775 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:39.168357 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" event={"ID":"4229762d-c3ea-43fd-851d-506bba6c76db","Type":"ContainerStarted","Data":"255da76bd72849965391c6da5931ad7a7277ba25face80270fffe3c74a2d84d2"} Oct 30 21:16:39.168775 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:39.168660 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:39.168775 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:39.168732 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:39.184831 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:39.184809 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:16:39.292512 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:39.292417 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" podStartSLOduration=9.054356643 podStartE2EDuration="24.292403185s" totalImagesPullingTime="15.238046542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:39.291731914 +0000 UTC m=+24.851071976" watchObservedRunningTime="2026-10-30 21:16:39.292403185 +0000 UTC m=+24.851743252" Oct 30 21:16:40.019709 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:40.019667 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h8c58"] Oct 30 21:16:40.019914 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:40.019823 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:40.019992 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:40.019944 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:40.022789 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:40.022753 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5vztw"] Oct 30 21:16:40.022938 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:40.022895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:40.023078 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:40.023053 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:40.023566 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:40.023548 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzs5d"] Oct 30 21:16:40.023646 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:40.023635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:40.023723 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:40.023710 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:41.972506 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:41.972473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:41.972902 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:41.972579 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:41.972902 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:41.972584 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:41.972902 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:41.972608 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:41.972902 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:41.972683 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:41.972902 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:41.972745 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:43.971993 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:43.971962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:43.972399 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:43.972019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:43.972399 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:43.972106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:43.972399 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:43.972105 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:43.972399 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:43.972195 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:43.972399 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:43.972266 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:45.972353 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:45.972319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:45.972738 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:45.972413 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:45.972738 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:45.972418 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:45.972738 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:45.972442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:45.972738 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:45.972525 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:45.972738 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:45.972600 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:47.971768 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:47.971734 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:47.972189 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:47.971790 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:47.972189 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:47.971802 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:47.972189 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:47.971862 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:47.972189 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:47.971943 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:47.972189 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:47.972025 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:48.690136 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:48.690097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:48.690436 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.690253 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:48.690436 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.690334 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs podName:12073275-f042-497b-97cc-10cf8d661a68 nodeName:}" failed. No retries permitted until 2026-10-30 21:17:20.690312084 +0000 UTC m=+66.249652135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs") pod "network-metrics-daemon-kzs5d" (UID: "12073275-f042-497b-97cc-10cf8d661a68") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:16:48.790824 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:48.790789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:48.791021 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:48.790859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:48.791021 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.790975 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:48.791021 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.790986 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:16:48.791021 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.791013 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:16:48.791021 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.791023 2572 projected.go:196] Error preparing data for projected volume kube-api-access-j4c44 for pod openshift-network-diagnostics/network-check-target-h8c58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:48.791178 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.791031 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret podName:21dcf34f-5308-47fd-a7a2-5a277aa95daf nodeName:}" failed. No retries permitted until 2026-10-30 21:17:20.791016032 +0000 UTC m=+66.350356081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret") pod "global-pull-secret-syncer-5vztw" (UID: "21dcf34f-5308-47fd-a7a2-5a277aa95daf") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:16:48.791178 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:48.791062 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44 podName:aa327463-70c6-4971-9554-b2df6bce21e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:17:20.791050667 +0000 UTC m=+66.350390721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j4c44" (UniqueName: "kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44") pod "network-check-target-h8c58" (UID: "aa327463-70c6-4971-9554-b2df6bce21e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:16:49.933763 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:49.933736 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-67mbf_1f0fea2c-ba27-4a1c-a48d-c070d9577b1b/dns-node-resolver/0.log" Oct 30 21:16:49.971889 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:49.971862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:49.972028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:49.971906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:49.972028 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:49.971994 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:49.972114 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:49.972044 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:49.972154 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:49.972123 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:49.972193 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:49.972180 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:50.748031 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:50.748004 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5bz2b_e2a1b8a3-1ae3-4724-ba2e-7260a79c6812/node-ca/0.log" Oct 30 21:16:51.972536 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:51.972501 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:51.972536 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:51.972536 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:51.972955 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:51.972593 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:51.972955 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:51.972641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:51.972955 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:51.972743 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:51.972955 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:51.972815 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:53.972434 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:53.972399 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:53.972434 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:53.972425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:53.972955 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:53.972503 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:53.972955 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:53.972537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:53.972955 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:53.972635 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:53.972955 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:53.972763 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:55.971748 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:55.971706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:55.971748 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:55.971750 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:55.972303 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:55.971706 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:55.972303 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:55.971838 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:55.972303 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:55.971931 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:55.972303 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:55.971989 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:57.972132 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:57.972100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:57.972132 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:57.972130 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:57.972657 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:57.972137 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:57.972657 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:57.972221 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:16:57.972657 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:57.972310 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:57.972657 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:57.972390 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:59.972528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:59.972496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:16:59.972937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:59.972496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:16:59.972937 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:16:59.972496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:16:59.972937 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:59.972658 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:16:59.972937 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:59.972742 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:16:59.972937 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:16:59.972800 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:17:01.972502 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:01.972446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:01.972502 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:01.972507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:01.973021 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:01.972469 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:01.973021 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:17:01.972583 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzs5d" podUID="12073275-f042-497b-97cc-10cf8d661a68" Oct 30 21:17:01.973021 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:17:01.972630 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8c58" podUID="aa327463-70c6-4971-9554-b2df6bce21e5" Oct 30 21:17:01.973449 ip-10-0-142-119 kubenswrapper[2572]: E1030 21:17:01.973288 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5vztw" podUID="21dcf34f-5308-47fd-a7a2-5a277aa95daf" Oct 30 21:17:03.753289 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.753259 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeReady" Oct 30 21:17:03.753684 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.753407 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:17:03.849531 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.849496 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qx48k"] Oct 30 21:17:03.851096 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.851080 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:03.857090 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.857066 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ndqrr"] Oct 30 21:17:03.858588 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.858571 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:03.874315 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.874290 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7bckf"] Oct 30 21:17:03.875842 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.875822 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:03.903732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.903701 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:17:03.904214 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.904193 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:17:03.904339 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.904210 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42q2c\"" Oct 30 21:17:03.910015 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.909989 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qx48k"] Oct 30 21:17:03.917723 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.917696 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ndqrr"] Oct 30 21:17:03.972160 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.972070 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bckf"] Oct 30 21:17:03.972320 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.972268 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:03.972398 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.972372 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:03.972527 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.972503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:03.995833 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.995811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2fccb02-6982-45a8-9d67-fe60097a84c3-metrics-tls\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:03.995958 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.995840 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/720074df-0a6f-4778-bb73-1cd424158dc9-data-volume\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:03.995958 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.995860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nd2\" (UniqueName: \"kubernetes.io/projected/e2fccb02-6982-45a8-9d67-fe60097a84c3-kube-api-access-n9nd2\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:03.995958 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.995895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdn5q\" (UniqueName: \"kubernetes.io/projected/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-kube-api-access-bdn5q\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:03.996054 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.995957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/720074df-0a6f-4778-bb73-1cd424158dc9-crio-socket\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:03.996054 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.995984 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95n8\" (UniqueName: \"kubernetes.io/projected/720074df-0a6f-4778-bb73-1cd424158dc9-kube-api-access-w95n8\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:03.996054 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.996045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/720074df-0a6f-4778-bb73-1cd424158dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:03.996143 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.996064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/720074df-0a6f-4778-bb73-1cd424158dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:03.996143 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.996094 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fccb02-6982-45a8-9d67-fe60097a84c3-tmp-dir\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:03.996143 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.996122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2fccb02-6982-45a8-9d67-fe60097a84c3-config-volume\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:03.996228 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:03.996162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-cert\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004794 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004803 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004817 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004819 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004862 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004873 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:17:04.004815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.004896 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7dj2\"" Oct 30 21:17:04.005171 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.005020 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:17:04.005171 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.005025 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wvjk\"" Oct 30 21:17:04.097010 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.096967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2fccb02-6982-45a8-9d67-fe60097a84c3-metrics-tls\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.097010 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/720074df-0a6f-4778-bb73-1cd424158dc9-data-volume\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097221 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nd2\" (UniqueName: \"kubernetes.io/projected/e2fccb02-6982-45a8-9d67-fe60097a84c3-kube-api-access-n9nd2\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.097221 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097047 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn5q\" (UniqueName: \"kubernetes.io/projected/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-kube-api-access-bdn5q\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:04.097221 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/720074df-0a6f-4778-bb73-1cd424158dc9-crio-socket\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097338 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097224 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95n8\" (UniqueName: \"kubernetes.io/projected/720074df-0a6f-4778-bb73-1cd424158dc9-kube-api-access-w95n8\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097338 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/720074df-0a6f-4778-bb73-1cd424158dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097338 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/720074df-0a6f-4778-bb73-1cd424158dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097338 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fccb02-6982-45a8-9d67-fe60097a84c3-tmp-dir\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.097528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097371 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2fccb02-6982-45a8-9d67-fe60097a84c3-config-volume\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.097528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097387 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/720074df-0a6f-4778-bb73-1cd424158dc9-data-volume\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097400 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/720074df-0a6f-4778-bb73-1cd424158dc9-crio-socket\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.097528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-cert\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:04.097786 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.097677 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fccb02-6982-45a8-9d67-fe60097a84c3-tmp-dir\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.098026 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.098003 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2fccb02-6982-45a8-9d67-fe60097a84c3-config-volume\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.098220 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.098203 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/720074df-0a6f-4778-bb73-1cd424158dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.101050 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.101026 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2fccb02-6982-45a8-9d67-fe60097a84c3-metrics-tls\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.101200 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.101185 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/720074df-0a6f-4778-bb73-1cd424158dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.101455 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.101436 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-cert\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:04.104981 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.104959 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:17:04.105087 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.104959 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:17:04.105087 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.105024 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:17:04.105184 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.105172 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:17:04.105184 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.105174 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lps2\"" Oct 30 21:17:04.105275 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.105228 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p2m87\"" Oct 30 21:17:04.105275 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.105232 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5jmhs\"" Oct 30 21:17:04.153640 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.153608 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95n8\" (UniqueName: \"kubernetes.io/projected/720074df-0a6f-4778-bb73-1cd424158dc9-kube-api-access-w95n8\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.153777 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.153610 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nd2\" (UniqueName: \"kubernetes.io/projected/e2fccb02-6982-45a8-9d67-fe60097a84c3-kube-api-access-n9nd2\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.153867 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.153834 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn5q\" (UniqueName: \"kubernetes.io/projected/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-kube-api-access-bdn5q\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:04.162654 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.162633 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:04.168805 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.168782 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:04.185990 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.185964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:04.548221 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.548145 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ndqrr"] Oct 30 21:17:04.551714 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:04.551690 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6a500a_0ebe_4a44_ba07_8b33ad54c2dd.slice/crio-db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508 WatchSource:0}: Error finding container db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508: Status 404 returned error can't find the container with id db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508 Oct 30 21:17:04.567522 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.567486 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qx48k"] Oct 30 21:17:04.570705 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:04.570679 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fccb02_6982_45a8_9d67_fe60097a84c3.slice/crio-83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa WatchSource:0}: Error finding container 83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa: Status 404 returned error can't find the container with id 83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa Oct 30 21:17:04.597621 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:04.597594 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7bckf"] Oct 30 21:17:04.600694 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:04.600671 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720074df_0a6f_4778_bb73_1cd424158dc9.slice/crio-5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621 WatchSource:0}: Error finding container 5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621: Status 404 returned error can't find the container with id 5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621 Oct 30 21:17:05.232020 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:05.231986 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bckf" event={"ID":"720074df-0a6f-4778-bb73-1cd424158dc9","Type":"ContainerStarted","Data":"f8549b9e876beb8aae858959559efc486a05ab623375e64cf976a4633ea89928"} Oct 30 21:17:05.232485 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:05.232074 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bckf" event={"ID":"720074df-0a6f-4778-bb73-1cd424158dc9","Type":"ContainerStarted","Data":"5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621"} Oct 30 21:17:05.233128 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:05.233095 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ndqrr" event={"ID":"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd","Type":"ContainerStarted","Data":"db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508"} Oct 30 21:17:05.234254 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:05.234230 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qx48k" event={"ID":"e2fccb02-6982-45a8-9d67-fe60097a84c3","Type":"ContainerStarted","Data":"83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa"} Oct 30 21:17:06.240619 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:06.240572 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bckf" event={"ID":"720074df-0a6f-4778-bb73-1cd424158dc9","Type":"ContainerStarted","Data":"3d39d11313d4a8c6acd10d5ac3f925860f49817a4f0f249d61fc8ef685efce2f"} Oct 30 21:17:07.245801 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:07.245708 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qx48k" event={"ID":"e2fccb02-6982-45a8-9d67-fe60097a84c3","Type":"ContainerStarted","Data":"d5f2f3c4b45ed5c80da38da56101d848cb008f58c377377a2987df57b30cf7cf"} Oct 30 21:17:07.245801 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:07.245771 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qx48k" event={"ID":"e2fccb02-6982-45a8-9d67-fe60097a84c3","Type":"ContainerStarted","Data":"0a0e31c86200b37f771f70eeffc31bf635bc6f4bc6eb8cc5f56d64e1d8718dd6"} Oct 30 21:17:07.246273 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:07.245852 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:07.989689 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:07.989624 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-qx48k" podStartSLOduration=2.673325803 podStartE2EDuration="4.989604565s" totalImagesPullingTime="2.316278762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:07.3601457 +0000 UTC m=+52.919485752" watchObservedRunningTime="2026-10-30 21:17:07.989604565 +0000 UTC m=+53.548944622" Oct 30 21:17:07.989854 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:07.989798 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4"] Oct 30 21:17:07.992858 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:07.992843 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.094194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.094105 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:17:08.094194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.094133 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:17:08.094194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.094176 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:17:08.094194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.094176 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:17:08.094411 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.094196 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:17:08.094411 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.094212 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-crhr9\"" Oct 30 21:17:08.095542 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.095524 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4"] Oct 30 21:17:08.131919 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.131875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.132067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.132029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4mj8\" (UniqueName: \"kubernetes.io/projected/2bfc67d0-5890-47b5-9885-4338121de31e-kube-api-access-k4mj8\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.132105 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.132078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.132105 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.132095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc67d0-5890-47b5-9885-4338121de31e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.232927 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.232893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.232927 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.232931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc67d0-5890-47b5-9885-4338121de31e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.233206 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.232958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.233206 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.233104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mj8\" (UniqueName: \"kubernetes.io/projected/2bfc67d0-5890-47b5-9885-4338121de31e-kube-api-access-k4mj8\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.233735 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.233656 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc67d0-5890-47b5-9885-4338121de31e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.235297 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.235276 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.235384 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.235332 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.249432 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.249403 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ndqrr" event={"ID":"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd","Type":"ContainerStarted","Data":"a8f9612d7400b9298480bcb893cb7f573373a2afd11e7d9db36d3db10aaaa2e2"} Oct 30 21:17:08.251362 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.251336 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7bckf" event={"ID":"720074df-0a6f-4778-bb73-1cd424158dc9","Type":"ContainerStarted","Data":"3d5a6048c3365451773778a5797de53b57b85a402acf2921e02127fed9b56d60"} Oct 30 21:17:08.289509 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.289478 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mj8\" (UniqueName: \"kubernetes.io/projected/2bfc67d0-5890-47b5-9885-4338121de31e-kube-api-access-k4mj8\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.306667 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.306641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:08.363299 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.363254 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ndqrr" podStartSLOduration=2.069202677 podStartE2EDuration="5.363239988s" totalImagesPullingTime="3.294037311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:08.362481512 +0000 UTC m=+53.921821581" watchObservedRunningTime="2026-10-30 21:17:08.363239988 +0000 UTC m=+53.922580040" Oct 30 21:17:08.478053 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.477996 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7bckf" podStartSLOduration=2.450432496 podStartE2EDuration="5.477978128s" totalImagesPullingTime="3.027545632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:17:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:08.477116354 +0000 UTC m=+54.036456406" watchObservedRunningTime="2026-10-30 21:17:08.477978128 +0000 UTC m=+54.037318181" Oct 30 21:17:08.673007 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:08.672974 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4"] Oct 30 21:17:08.676186 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:08.676159 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfc67d0_5890_47b5_9885_4338121de31e.slice/crio-13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33 WatchSource:0}: Error finding container 13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33: Status 404 returned error can't find the container with id 13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33 Oct 30 21:17:09.264060 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:09.264023 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" event={"ID":"2bfc67d0-5890-47b5-9885-4338121de31e","Type":"ContainerStarted","Data":"13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33"} Oct 30 21:17:11.183671 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:11.183645 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:11.271951 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:11.271909 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" event={"ID":"2bfc67d0-5890-47b5-9885-4338121de31e","Type":"ContainerStarted","Data":"00e9cd43a50f6a86689d3085d41ba37c25f481f55c975a3a665010d5441d3514"} Oct 30 21:17:11.272132 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:11.271963 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" event={"ID":"2bfc67d0-5890-47b5-9885-4338121de31e","Type":"ContainerStarted","Data":"a2376ac7af2ae5af8176535fc625d14be92b6cc288134567beef66effd685a80"} Oct 30 21:17:11.518453 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:11.518390 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" podStartSLOduration=2.104270599 podStartE2EDuration="4.518370102s" totalImagesPullingTime="2.414099503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:11.517221955 +0000 UTC m=+57.076562007" watchObservedRunningTime="2026-10-30 21:17:11.518370102 +0000 UTC m=+57.077710153" Oct 30 21:17:14.456417 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.456384 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z"] Oct 30 21:17:14.459581 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.459563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.562834 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.562791 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:17:14.563916 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.563274 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:17:14.563916 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.563360 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fgt7b\"" Oct 30 21:17:14.565138 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.565109 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z"] Oct 30 21:17:14.576969 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.576935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.577099 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.576990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3877d90b-6949-48e6-bdd5-568a248a31d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.577099 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.577009 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7tz\" (UniqueName: \"kubernetes.io/projected/3877d90b-6949-48e6-bdd5-568a248a31d9-kube-api-access-9x7tz\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.577169 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.577137 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.670311 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.670280 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll"] Oct 30 21:17:14.673340 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.673325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.675125 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.675100 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qz9gs"] Oct 30 21:17:14.677924 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.677910 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.678159 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.678113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.678247 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.678196 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.678303 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.678246 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3877d90b-6949-48e6-bdd5-568a248a31d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.678303 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.678270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7tz\" (UniqueName: \"kubernetes.io/projected/3877d90b-6949-48e6-bdd5-568a248a31d9-kube-api-access-9x7tz\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.679090 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.678998 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3877d90b-6949-48e6-bdd5-568a248a31d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.680915 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.680893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.681015 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.680997 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.725141 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.725056 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:17:14.725141 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.725108 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:17:14.725348 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.725212 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:17:14.725348 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.725218 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8tbzb\"" Oct 30 21:17:14.729110 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.729094 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:17:14.729211 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.729141 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:17:14.729211 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.729134 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:17:14.729284 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.729218 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qn5sj\"" Oct 30 21:17:14.734757 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.734738 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7tz\" (UniqueName: \"kubernetes.io/projected/3877d90b-6949-48e6-bdd5-568a248a31d9-kube-api-access-9x7tz\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.773739 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.773713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:14.774910 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.774871 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll"] Oct 30 21:17:14.778923 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.778899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.778988 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.778949 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-root\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779053 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-wtmp\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779089 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779149 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxqf\" (UniqueName: \"kubernetes.io/projected/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-api-access-9rxqf\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.779212 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.779251 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.779282 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-textfile\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779342 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779331 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-sys\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779388 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779351 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-metrics-client-ca\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779388 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbvb\" (UniqueName: \"kubernetes.io/projected/985581c6-d015-4770-992d-b75ab7f0654a-kube-api-access-4dbvb\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779448 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.779448 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779429 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.779517 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.779619 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.779594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-tls\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.880708 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.880708 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-root\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.880959 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-wtmp\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.880959 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.880959 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880823 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-root\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.880959 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxqf\" (UniqueName: \"kubernetes.io/projected/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-api-access-9rxqf\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.880959 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880917 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-wtmp\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.880969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-textfile\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881192 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-sys\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-metrics-client-ca\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbvb\" (UniqueName: \"kubernetes.io/projected/985581c6-d015-4770-992d-b75ab7f0654a-kube-api-access-4dbvb\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881317 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881333 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-sys\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.881414 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.882120 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.882120 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881489 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-tls\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.882120 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881559 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-textfile\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.882120 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.881860 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.882279 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.882123 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.882279 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.882251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-metrics-client-ca\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.919167 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.919142 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75748b998b-p9kst"] Oct 30 21:17:14.923581 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.923561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:14.932235 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.932205 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:17:14.932398 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.932381 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:17:14.932813 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.932794 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:17:14.932964 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.932942 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:17:14.933126 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.933100 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:17:14.933126 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.933109 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:17:14.938667 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.938645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbvb\" (UniqueName: \"kubernetes.io/projected/985581c6-d015-4770-992d-b75ab7f0654a-kube-api-access-4dbvb\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.938975 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.938958 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxqf\" (UniqueName: \"kubernetes.io/projected/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-api-access-9rxqf\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.942259 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.942235 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.942372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.942269 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.944028 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.944007 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-tls\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.944737 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.944454 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:14.944737 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.944701 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.944737 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.944729 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:14.975653 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975591 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:17:14.975653 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975638 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:17:14.975814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975693 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:17:14.975814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975697 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:17:14.975814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975713 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:17:14.975814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975714 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9jlcd\"" Oct 30 21:17:14.975814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975736 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:17:14.975814 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.975739 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:17:14.982101 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:14.982070 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75748b998b-p9kst"] Oct 30 21:17:15.039576 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.039550 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8tbzb\"" Oct 30 21:17:15.049744 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.049722 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:15.050226 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.050210 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qn5sj\"" Oct 30 21:17:15.060845 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.060825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:15.071730 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:15.071702 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985581c6_d015_4770_992d_b75ab7f0654a.slice/crio-c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1 WatchSource:0}: Error finding container c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1: Status 404 returned error can't find the container with id c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1 Oct 30 21:17:15.082400 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.082574 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082418 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.082574 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.082574 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.082760 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082606 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.082760 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.082760 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.082697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.148582 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.148460 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z"] Oct 30 21:17:15.150921 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:15.150895 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877d90b_6949_48e6_bdd5_568a248a31d9.slice/crio-afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a WatchSource:0}: Error finding container afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a: Status 404 returned error can't find the container with id afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a Oct 30 21:17:15.183334 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.183474 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.183474 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183384 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.183474 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.183474 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.183474 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183450 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.183720 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.183617 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.184548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.184229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.184548 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.184495 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.184832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.184814 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.185253 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.185234 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.186349 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.186332 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.186772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.186756 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.240684 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.240657 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.285589 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.285551 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz9gs" event={"ID":"985581c6-d015-4770-992d-b75ab7f0654a","Type":"ContainerStarted","Data":"c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1"} Oct 30 21:17:15.287251 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.287224 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" event={"ID":"3877d90b-6949-48e6-bdd5-568a248a31d9","Type":"ContainerStarted","Data":"d62f3fe4cbecdd920843d0e4473b4a6d4b3e8a897fda103190dd1c209aa709fe"} Oct 30 21:17:15.287372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.287282 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" event={"ID":"3877d90b-6949-48e6-bdd5-568a248a31d9","Type":"ContainerStarted","Data":"88903123dc6c554e5536b7557580b8f194de11a75a849cefa3de46a520a90810"} Oct 30 21:17:15.287372 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.287309 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" event={"ID":"3877d90b-6949-48e6-bdd5-568a248a31d9","Type":"ContainerStarted","Data":"afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a"} Oct 30 21:17:15.420316 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.420288 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll"] Oct 30 21:17:15.422957 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:15.422931 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb13caf_5f94_4a1a_a865_8f3a1ee584fe.slice/crio-36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718 WatchSource:0}: Error finding container 36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718: Status 404 returned error can't find the container with id 36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718 Oct 30 21:17:15.536100 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.536020 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:15.902737 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:15.902697 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75748b998b-p9kst"] Oct 30 21:17:15.906702 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:15.906675 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ab801c_ab61_43bb_8523_8798c89b5279.slice/crio-64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee WatchSource:0}: Error finding container 64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee: Status 404 returned error can't find the container with id 64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee Oct 30 21:17:16.293699 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.293622 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75748b998b-p9kst" event={"ID":"39ab801c-ab61-43bb-8523-8798c89b5279","Type":"ContainerStarted","Data":"64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee"} Oct 30 21:17:16.295070 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.295048 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" event={"ID":"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe","Type":"ContainerStarted","Data":"36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718"} Oct 30 21:17:16.809753 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.809722 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:17:16.816632 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.816469 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.871348 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.871270 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872174 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872171 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872192 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872205 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872205 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872211 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872292 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:17:16.872367 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.872349 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-57txw\"" Oct 30 21:17:16.973788 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.973723 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:17:16.998194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998158 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998194 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998399 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998212 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998399 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998301 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998510 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998510 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998605 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998605 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998605 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998605 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998785 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998611 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998785 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998633 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:16.998785 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:16.998651 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.021635 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.021602 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:17:17.099847 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.099847 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.099847 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.099847 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.099956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.100004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.100026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.100048 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.100070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.100088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.100164 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.100122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.101827 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.101475 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.101827 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.101747 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.102476 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.102447 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.107021 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.106997 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.108576 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.108554 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.108667 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.108639 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.108743 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.108667 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.108978 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.108864 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.109061 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.108952 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.109061 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.109052 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.109381 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.109343 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.110635 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.110599 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.170573 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.170471 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.253409 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.253376 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:17.294509 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.294478 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b"] Oct 30 21:17:17.298099 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.298075 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.301788 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.301238 2572 generic.go:356] "Generic (PLEG): container finished" podID="985581c6-d015-4770-992d-b75ab7f0654a" containerID="b52bf282cf583348698dcc5487aaf6e2b272e468b2a21446e5661ed863308f4c" exitCode=0 Oct 30 21:17:17.301788 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.301393 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz9gs" event={"ID":"985581c6-d015-4770-992d-b75ab7f0654a","Type":"ContainerDied","Data":"b52bf282cf583348698dcc5487aaf6e2b272e468b2a21446e5661ed863308f4c"} Oct 30 21:17:17.304649 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.304516 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" event={"ID":"3877d90b-6949-48e6-bdd5-568a248a31d9","Type":"ContainerStarted","Data":"2c68f41bb67f2d4a3da2dfe45fc9d1f2aab7e56036318031df600f527715cf5d"} Oct 30 21:17:17.350832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350663 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:17:17.350832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350663 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:17:17.350832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350672 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a115kei9rjd2d\"" Oct 30 21:17:17.350832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350721 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:17:17.350832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350776 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nxjfg\"" Oct 30 21:17:17.350832 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350786 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:17:17.351219 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.350856 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:17:17.368848 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.368652 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b"] Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402232 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402381 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e7aad03-570e-43d0-bde7-83dd1740ac78-metrics-client-ca\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e7aad03-570e-43d0-bde7-83dd1740ac78-tmp\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-grpc-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd62m\" (UniqueName: \"kubernetes.io/projected/7e7aad03-570e-43d0-bde7-83dd1740ac78-kube-api-access-gd62m\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.402600 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.402545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.446909 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.446794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:17.487528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.486729 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" podStartSLOduration=1.855626051 podStartE2EDuration="3.486712419s" totalImagesPullingTime="1.631086368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:17.486135967 +0000 UTC m=+63.045476032" watchObservedRunningTime="2026-10-30 21:17:17.486712419 +0000 UTC m=+63.046052471" Oct 30 21:17:17.503560 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.503527 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.520736 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.520681 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.521018 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.520994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.521662 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.521640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.522299 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.522279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.522694 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.522680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.524624 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.523208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e7aad03-570e-43d0-bde7-83dd1740ac78-metrics-client-ca\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.524624 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.523270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e7aad03-570e-43d0-bde7-83dd1740ac78-tmp\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.524624 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.523330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-grpc-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.524624 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.523359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd62m\" (UniqueName: \"kubernetes.io/projected/7e7aad03-570e-43d0-bde7-83dd1740ac78-kube-api-access-gd62m\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.524624 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.524350 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e7aad03-570e-43d0-bde7-83dd1740ac78-metrics-client-ca\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.524624 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.524582 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e7aad03-570e-43d0-bde7-83dd1740ac78-tmp\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.527507 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.527460 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.528116 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.528084 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.528351 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.528327 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-grpc-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.539497 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.539443 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.540303 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.540269 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.579858 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.579826 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd62m\" (UniqueName: \"kubernetes.io/projected/7e7aad03-570e-43d0-bde7-83dd1740ac78-kube-api-access-gd62m\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.625893 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.625855 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:17.891235 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:17.891020 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:17:17.898654 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:17.898622 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e92bbc_f24d_4a92_b28f_496dbeef61ce.slice/crio-ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c WatchSource:0}: Error finding container ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c: Status 404 returned error can't find the container with id ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c Oct 30 21:17:18.003784 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.003749 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b"] Oct 30 21:17:18.010644 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:18.010603 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7aad03_570e_43d0_bde7_83dd1740ac78.slice/crio-2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f WatchSource:0}: Error finding container 2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f: Status 404 returned error can't find the container with id 2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f Oct 30 21:17:18.310346 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.310256 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f"} Oct 30 21:17:18.311688 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.311661 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c"} Oct 30 21:17:18.314365 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.314343 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz9gs" event={"ID":"985581c6-d015-4770-992d-b75ab7f0654a","Type":"ContainerStarted","Data":"7e7f38d4074f083bb14e9ba23fe630493642dd2bfbd854d31f3dd02cf75008d3"} Oct 30 21:17:18.314455 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.314390 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qz9gs" event={"ID":"985581c6-d015-4770-992d-b75ab7f0654a","Type":"ContainerStarted","Data":"c144ea28c3cb37fbd02038e41878b985b8c049831275be24d3ed44ec54b40f15"} Oct 30 21:17:18.316934 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.316909 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" event={"ID":"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe","Type":"ContainerStarted","Data":"c511ad575d05bb24a98339a76c1c16615d61c1932166d2bc598f0120c6b4ac54"} Oct 30 21:17:18.317037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.316959 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" event={"ID":"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe","Type":"ContainerStarted","Data":"b17c61707dd78fe96d55013be16e9953b64b48054b4d31b938f48dca70f69d27"} Oct 30 21:17:18.317037 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.316991 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" event={"ID":"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe","Type":"ContainerStarted","Data":"2b9003c41a658c54419416fb4ccd764bce466e41edd48e83624fd87b524efa41"} Oct 30 21:17:18.429767 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.429504 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qz9gs" podStartSLOduration=2.848012319 podStartE2EDuration="4.429485351s" totalImagesPullingTime="1.490918957s" totalInitContainerRuntime="90.554075ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:18.4292422 +0000 UTC m=+63.988582254" watchObservedRunningTime="2026-10-30 21:17:18.429485351 +0000 UTC m=+63.988825404" Oct 30 21:17:18.546071 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:18.546013 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" podStartSLOduration=2.722331718 podStartE2EDuration="4.54599099s" totalImagesPullingTime="1.823659272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:18.544542414 +0000 UTC m=+64.103882465" watchObservedRunningTime="2026-10-30 21:17:18.54599099 +0000 UTC m=+64.105331043" Oct 30 21:17:19.297030 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.295796 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f"] Oct 30 21:17:19.300528 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.300503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.341907 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.341352 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/996da354-a8e9-4600-9cad-e8beba46a6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.341907 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.341514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7pb\" (UniqueName: \"kubernetes.io/projected/996da354-a8e9-4600-9cad-e8beba46a6ab-kube-api-access-wn7pb\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.355212 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.355142 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b4tkk\"" Oct 30 21:17:19.356057 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.356004 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:17:19.357701 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.357639 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f"] Oct 30 21:17:19.442929 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.442862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7pb\" (UniqueName: \"kubernetes.io/projected/996da354-a8e9-4600-9cad-e8beba46a6ab-kube-api-access-wn7pb\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.443070 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.442998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/996da354-a8e9-4600-9cad-e8beba46a6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.446784 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.446730 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/996da354-a8e9-4600-9cad-e8beba46a6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.496945 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.496912 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dcc758dc7-djcnn"] Oct 30 21:17:19.501093 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.500980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.504685 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.504234 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7pb\" (UniqueName: \"kubernetes.io/projected/996da354-a8e9-4600-9cad-e8beba46a6ab-kube-api-access-wn7pb\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.543556 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.543686 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543581 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-metrics-server-audit-profiles\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.543686 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543609 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-client-certs\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.543793 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-client-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.543793 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk4dz\" (UniqueName: \"kubernetes.io/projected/7496f3c4-d42d-446d-8c0e-32272aa84f34-kube-api-access-tk4dz\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.543936 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-tls\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.543994 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.543958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7496f3c4-d42d-446d-8c0e-32272aa84f34-audit-log\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.553397 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.553555 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.553633 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.553689 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-84xd8\"" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.553773 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.553828 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3l2gij8kn99ct\"" Oct 30 21:17:19.559239 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.558627 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dcc758dc7-djcnn"] Oct 30 21:17:19.617024 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.616994 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:19.645555 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-tls\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.645717 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7496f3c4-d42d-446d-8c0e-32272aa84f34-audit-log\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.645717 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.645717 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-metrics-server-audit-profiles\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.645717 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-client-certs\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.646014 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-client-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.646014 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.645750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4dz\" (UniqueName: \"kubernetes.io/projected/7496f3c4-d42d-446d-8c0e-32272aa84f34-kube-api-access-tk4dz\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.647346 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.647318 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7496f3c4-d42d-446d-8c0e-32272aa84f34-audit-log\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.647684 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.647348 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-metrics-server-audit-profiles\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.647967 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.647920 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.649088 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.649039 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-tls\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.650159 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.650134 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-client-certs\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.650285 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.650266 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-client-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.702977 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.702627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4dz\" (UniqueName: \"kubernetes.io/projected/7496f3c4-d42d-446d-8c0e-32272aa84f34-kube-api-access-tk4dz\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.844276 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.844199 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:19.999626 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:19.999522 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f"] Oct 30 21:17:20.002372 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:20.002340 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996da354_a8e9_4600_9cad_e8beba46a6ab.slice/crio-2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d WatchSource:0}: Error finding container 2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d: Status 404 returned error can't find the container with id 2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d Oct 30 21:17:20.215119 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.215076 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8dcd9788c-w4vk7"] Oct 30 21:17:20.218779 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.218746 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dcc758dc7-djcnn"] Oct 30 21:17:20.218921 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.218860 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.250640 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.250597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-serving-certs-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.250786 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.250749 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.250786 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.250775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.250861 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.250827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.250949 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.250866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s7f2\" (UniqueName: \"kubernetes.io/projected/0a04c0a7-b90d-4233-bc46-b8430095ef21-kube-api-access-6s7f2\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.251004 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.250954 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-metrics-client-ca\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.251047 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.251025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.251100 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.251060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-federate-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.269548 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:20.267092 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7496f3c4_d42d_446d_8c0e_32272aa84f34.slice/crio-7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d WatchSource:0}: Error finding container 7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d: Status 404 returned error can't find the container with id 7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d Oct 30 21:17:20.271280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.271150 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:17:20.271280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.271153 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:17:20.271280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.271231 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:17:20.271280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.271257 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r4rjf\"" Oct 30 21:17:20.271280 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.271259 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:17:20.271593 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.271304 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:17:20.321484 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.321436 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:17:20.322783 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.322736 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8dcd9788c-w4vk7"] Oct 30 21:17:20.327804 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.327774 2572 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" exitCode=0 Oct 30 21:17:20.327971 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.327865 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} Oct 30 21:17:20.329212 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.329163 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" event={"ID":"7496f3c4-d42d-446d-8c0e-32272aa84f34","Type":"ContainerStarted","Data":"7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d"} Oct 30 21:17:20.331110 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.331082 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75748b998b-p9kst" event={"ID":"39ab801c-ab61-43bb-8523-8798c89b5279","Type":"ContainerStarted","Data":"52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340"} Oct 30 21:17:20.332559 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.332539 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" event={"ID":"996da354-a8e9-4600-9cad-e8beba46a6ab","Type":"ContainerStarted","Data":"2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d"} Oct 30 21:17:20.351589 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.351558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.351715 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.351594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s7f2\" (UniqueName: \"kubernetes.io/projected/0a04c0a7-b90d-4233-bc46-b8430095ef21-kube-api-access-6s7f2\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.351715 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.351651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-metrics-client-ca\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.351815 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.351763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.351923 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.351876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-federate-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.351993 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.351941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-serving-certs-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.353175 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.352800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.353175 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.352896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.353175 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.352926 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-serving-certs-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.353398 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.353262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-metrics-client-ca\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.353648 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.353619 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.354692 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.354667 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.354863 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.354839 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-federate-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.355772 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.355752 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.356114 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.356094 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.408855 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.408824 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s7f2\" (UniqueName: \"kubernetes.io/projected/0a04c0a7-b90d-4233-bc46-b8430095ef21-kube-api-access-6s7f2\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.454141 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.452713 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75748b998b-p9kst" podStartSLOduration=2.8964320690000003 podStartE2EDuration="6.452692997s" totalImagesPullingTime="3.556260928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:20.451524361 +0000 UTC m=+66.010864413" watchObservedRunningTime="2026-10-30 21:17:20.452692997 +0000 UTC m=+66.012033047" Oct 30 21:17:20.536262 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.536173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:20.762833 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.762808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:20.813616 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.813593 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:17:20.826011 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.825985 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:20.864261 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.864206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:20.864428 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.864290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:20.915061 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.915036 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:17:20.915146 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.915074 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:17:20.927293 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.927265 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:20.975992 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.975958 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:17:20.977817 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.977734 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8dcd9788c-w4vk7"] Oct 30 21:17:20.979708 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:20.979684 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a04c0a7_b90d_4233_bc46_b8430095ef21.slice/crio-36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b WatchSource:0}: Error finding container 36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b: Status 404 returned error can't find the container with id 36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b Oct 30 21:17:20.987591 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:20.987567 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:21.135057 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.135032 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lps2\"" Oct 30 21:17:21.140082 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.140058 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p2m87\"" Oct 30 21:17:21.144446 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.144423 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5jmhs\"" Oct 30 21:17:21.145425 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.145393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:21.150506 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.150490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:21.154667 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.154648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:21.341623 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.341518 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" event={"ID":"0a04c0a7-b90d-4233-bc46-b8430095ef21","Type":"ContainerStarted","Data":"36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b"} Oct 30 21:17:21.344999 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.344963 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"f4aca13134f3d5481ff58aaec4df8b10e874dd05acb50dea07fe74c3b9127bdb"} Oct 30 21:17:21.345132 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.345042 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"77965d4ddbd8fc60fb938cfcb0dcf0505a793e7b04b604e020cb3fec21cfbc2e"} Oct 30 21:17:21.345132 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.345084 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"4a80ab973c6ba12fad2a826502a4e992bc1a53d33b0680c06c24d0cd4dd11acb"} Oct 30 21:17:21.523214 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.523135 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzs5d"] Oct 30 21:17:21.537455 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:21.531062 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12073275_f042_497b_97cc_10cf8d661a68.slice/crio-d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6 WatchSource:0}: Error finding container d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6: Status 404 returned error can't find the container with id d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6 Oct 30 21:17:21.548227 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.548201 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5vztw"] Oct 30 21:17:21.550919 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:21.550869 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dcf34f_5308_47fd_a7a2_5a277aa95daf.slice/crio-0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e WatchSource:0}: Error finding container 0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e: Status 404 returned error can't find the container with id 0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e Oct 30 21:17:21.551443 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:21.551403 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h8c58"] Oct 30 21:17:21.556728 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:21.556681 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa327463_70c6_4971_9554_b2df6bce21e5.slice/crio-6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768 WatchSource:0}: Error finding container 6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768: Status 404 returned error can't find the container with id 6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768 Oct 30 21:17:22.354733 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:22.354695 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5vztw" event={"ID":"21dcf34f-5308-47fd-a7a2-5a277aa95daf","Type":"ContainerStarted","Data":"0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e"} Oct 30 21:17:22.356755 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:22.356663 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h8c58" event={"ID":"aa327463-70c6-4971-9554-b2df6bce21e5","Type":"ContainerStarted","Data":"6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768"} Oct 30 21:17:22.360490 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:22.360417 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzs5d" event={"ID":"12073275-f042-497b-97cc-10cf8d661a68","Type":"ContainerStarted","Data":"d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6"} Oct 30 21:17:23.370987 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.369844 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" event={"ID":"996da354-a8e9-4600-9cad-e8beba46a6ab","Type":"ContainerStarted","Data":"f1cf58bf328dedcc21e22ae8aa51f526e7d42c54d98a22c9a8682d78ba141394"} Oct 30 21:17:23.370987 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.370865 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:23.376896 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.376808 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:23.378866 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.378727 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"dde1d472340fccca046dee1e0030d80ac69dce917de874f339866887705b16c8"} Oct 30 21:17:23.378866 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.378802 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"91d8c2429fca7d7205358189af5f63b170401b6951a16db207f3f8dc56cfec90"} Oct 30 21:17:23.378866 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.378840 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" event={"ID":"7e7aad03-570e-43d0-bde7-83dd1740ac78","Type":"ContainerStarted","Data":"baf9a15f83164df07c7725fe13567b1796c9ca272b71dedf1246c76c9e509410"} Oct 30 21:17:23.379102 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.378896 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:23.385228 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.385042 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} Oct 30 21:17:23.385228 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.385147 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} Oct 30 21:17:23.385228 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.385188 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} Oct 30 21:17:23.385228 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.385228 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} Oct 30 21:17:23.385501 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.385266 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} Oct 30 21:17:23.385501 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.385305 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerStarted","Data":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} Oct 30 21:17:23.389905 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.389716 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" event={"ID":"7496f3c4-d42d-446d-8c0e-32272aa84f34","Type":"ContainerStarted","Data":"ebc0ebb0151e06f6aa08e31d2d2c1fa261f811c2970468060b7f1d302cb2a9b0"} Oct 30 21:17:23.484915 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.484376 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" podStartSLOduration=1.622689529 podStartE2EDuration="4.484361792s" totalImagesPullingTime="2.861672263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:23.482778512 +0000 UTC m=+69.042118566" watchObservedRunningTime="2026-10-30 21:17:23.484361792 +0000 UTC m=+69.043701843" Oct 30 21:17:23.606906 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.606805 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.562101895 podStartE2EDuration="7.606784233s" totalImagesPullingTime="4.033452042s" totalInitContainerRuntime="11.230296ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:23.605019233 +0000 UTC m=+69.164359288" watchObservedRunningTime="2026-10-30 21:17:23.606784233 +0000 UTC m=+69.166124286" Oct 30 21:17:23.725082 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.725010 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" podStartSLOduration=2.611171697 podStartE2EDuration="6.72499085s" totalImagesPullingTime="4.113819153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:23.722226156 +0000 UTC m=+69.281566212" watchObservedRunningTime="2026-10-30 21:17:23.72499085 +0000 UTC m=+69.284330903" Oct 30 21:17:23.838099 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:23.838043 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" podStartSLOduration=2.440069097 podStartE2EDuration="4.838024275s" totalImagesPullingTime="2.397955178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:23.836677265 +0000 UTC m=+69.396017316" watchObservedRunningTime="2026-10-30 21:17:23.838024275 +0000 UTC m=+69.397364332" Oct 30 21:17:24.400740 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:24.400702 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzs5d" event={"ID":"12073275-f042-497b-97cc-10cf8d661a68","Type":"ContainerStarted","Data":"ede4f72bbea32814a2a4e3051937fd27634731ccdcc2e5f2d1d118b01f234aa0"} Oct 30 21:17:24.401192 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:24.400769 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzs5d" event={"ID":"12073275-f042-497b-97cc-10cf8d661a68","Type":"ContainerStarted","Data":"a059b6f30f10ab8b1bde73d6bec05dd6217f9519ba18586bfd178736c5619d09"} Oct 30 21:17:24.424549 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:24.424516 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" event={"ID":"0a04c0a7-b90d-4233-bc46-b8430095ef21","Type":"ContainerStarted","Data":"1862fdfb36e58c8927677410c29b516dd70aa4069d40b6f06c8623ea07927b86"} Oct 30 21:17:24.524110 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:24.522867 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kzs5d" podStartSLOduration=67.338393437 podStartE2EDuration="1m9.522850698s" totalImagesPullingTime="2.184457261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:24.522124541 +0000 UTC m=+70.081464609" watchObservedRunningTime="2026-10-30 21:17:24.522850698 +0000 UTC m=+70.082190750" Oct 30 21:17:25.431284 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:25.431240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" event={"ID":"0a04c0a7-b90d-4233-bc46-b8430095ef21","Type":"ContainerStarted","Data":"aec8d1944b772b63443229aafba83de5ae0ae5edd5f557fe5428270487c854cf"} Oct 30 21:17:25.431732 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:25.431317 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" event={"ID":"0a04c0a7-b90d-4233-bc46-b8430095ef21","Type":"ContainerStarted","Data":"09c9def4489dd7d5fdce71dc134f71a11b719a7d05f3fcbd41f95bacdeaa1541"} Oct 30 21:17:25.536515 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:25.536468 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:25.536801 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:25.536776 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:25.542966 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:25.542940 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:25.550029 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:25.549976 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" podStartSLOduration=2.228670229 podStartE2EDuration="5.5499606s" totalImagesPullingTime="3.321290371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:25.548671904 +0000 UTC m=+71.108011956" watchObservedRunningTime="2026-10-30 21:17:25.5499606 +0000 UTC m=+71.109300652" Oct 30 21:17:26.440501 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.440442 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h8c58" event={"ID":"aa327463-70c6-4971-9554-b2df6bce21e5","Type":"ContainerStarted","Data":"6b639e57750e38d8a6c5022aea3d75effd704dcff29ad425ed9af3bf7871966e"} Oct 30 21:17:26.440975 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.440536 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:26.445725 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.445697 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:26.553798 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.553742 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h8c58" podStartSLOduration=66.795425874 podStartE2EDuration="1m11.55372761s" totalImagesPullingTime="4.758301736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:26.551895922 +0000 UTC m=+72.111235977" watchObservedRunningTime="2026-10-30 21:17:26.55372761 +0000 UTC m=+72.113067662" Oct 30 21:17:26.690719 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.690687 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85894779cf-c48vt"] Oct 30 21:17:26.694365 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.694342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.796816 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.796782 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:17:26.824860 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.824820 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.824908 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.824938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.824984 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.825008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.825026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825067 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.825051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.825284 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.825076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.846927 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.846894 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85894779cf-c48vt"] Oct 30 21:17:26.925737 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.925694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.925955 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.925744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.925955 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.925824 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.925955 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.925861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926122 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.925983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926122 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926122 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926122 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926583 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926557 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926686 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926661 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.926764 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.927009 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.926964 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.927111 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.927087 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.929250 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.929230 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.929539 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.929515 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:26.983268 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:26.983197 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:27.008192 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:27.008150 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:27.390294 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:27.390261 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85894779cf-c48vt"] Oct 30 21:17:27.394345 ip-10-0-142-119 kubenswrapper[2572]: W1030 21:17:27.394316 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8caf36_f02d_46e0_a7f9_fec2987fbc0f.slice/crio-5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500 WatchSource:0}: Error finding container 5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500: Status 404 returned error can't find the container with id 5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500 Oct 30 21:17:27.449682 ip-10-0-142-119 kubenswrapper[2572]: I1030 21:17:27.449643 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85894779cf-c48vt" event={"ID":"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f","Type":"ContainerStarted","Data":"5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500"} Oct 30 21:17:28.035418 ip-10-0-142-119 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:17:28.043577 ip-10-0-142-119 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:17:28.043791 ip-10-0-142-119 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:17:28.043852 ip-10-0-142-119 systemd[1]: kubelet.service: Consumed 4.628s CPU time. Oct 30 21:17:28.054693 ip-10-0-142-119 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:17:28.129426 ip-10-0-142-119 kubenswrapper[5914]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:17:28.129426 ip-10-0-142-119 kubenswrapper[5914]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:17:28.129426 ip-10-0-142-119 kubenswrapper[5914]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:17:28.129426 ip-10-0-142-119 kubenswrapper[5914]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:17:28.131440 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131361 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.131440 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131441 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131445 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131450 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131453 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131456 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131459 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131461 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131464 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131467 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131469 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131472 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131475 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131477 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131480 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131482 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131485 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131487 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131490 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131493 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.131513 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131496 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131499 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131501 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131504 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131507 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131510 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131513 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131515 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131527 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131530 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131533 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131536 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131538 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131541 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131543 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131546 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131548 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131551 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131554 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.131972 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131556 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131559 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131562 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131564 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131567 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131569 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131572 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131574 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131577 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131580 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131582 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131585 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131587 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131590 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131592 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131596 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131599 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131601 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131604 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131607 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.132410 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131609 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131612 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131615 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131618 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131620 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131623 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131625 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131628 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131631 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131633 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131658 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131663 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131666 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131669 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131671 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131674 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131676 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131679 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131683 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131686 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.132932 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131689 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131691 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131694 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131697 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131700 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131703 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131705 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131708 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131711 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131713 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131716 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131718 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131721 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131723 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131726 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131732 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131735 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131738 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131740 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.133401 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131743 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131745 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131748 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131751 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131753 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131756 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131760 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.131765 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132520 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132529 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132533 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132537 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132540 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132544 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132547 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132549 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132552 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132555 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132558 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.133851 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132561 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132564 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132566 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132569 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132572 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132574 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132577 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132579 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132582 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132585 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132588 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132590 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132593 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132595 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132598 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132600 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132603 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132605 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132608 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132610 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.134327 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132613 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132615 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132617 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132620 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132623 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132626 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132628 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132631 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132634 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132636 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132639 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132642 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132644 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132647 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132649 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132652 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132654 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132657 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132659 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132662 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.134784 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132664 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132667 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132670 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132673 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132675 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132678 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132681 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132683 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132685 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132688 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132690 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132693 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132696 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132698 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132701 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132708 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132711 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132713 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132716 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132718 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.135273 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132721 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132724 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132726 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132729 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132732 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132734 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132737 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132739 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132742 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132744 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132746 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132749 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132752 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132754 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132757 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132759 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132763 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132765 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132768 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132770 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.136143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132773 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132775 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132778 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132780 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132783 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132785 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132788 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132792 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132797 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132800 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132803 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132805 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132808 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132811 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.132814 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133350 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133357 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133361 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.136777 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133364 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133368 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133372 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133375 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133380 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133383 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133386 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133389 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133392 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133394 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133398 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133400 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133403 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133406 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133409 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133412 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133415 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133417 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133420 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.137237 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133422 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133425 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133428 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133431 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133443 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133446 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133448 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133451 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133453 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133456 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133459 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133461 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133464 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133466 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133468 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133471 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133473 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133477 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133479 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133484 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.137698 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133487 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133490 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133494 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133497 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133500 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133503 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133506 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133509 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133511 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133514 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133516 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133519 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133521 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133524 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133526 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133529 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133531 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133534 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133536 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.138278 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133538 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133541 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133543 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133546 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133549 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133552 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133554 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133557 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133559 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133562 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133565 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133567 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133569 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133572 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133574 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133577 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133580 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133582 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133585 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133588 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.138800 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133591 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133593 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133596 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133599 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133601 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133603 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133606 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133608 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133611 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133613 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133615 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133618 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133620 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133624 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133628 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133630 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133633 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133636 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133638 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133641 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.139314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133643 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133646 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133648 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133651 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.133653 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133739 5914 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133747 5914 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133753 5914 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133757 5914 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133762 5914 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133765 5914 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133770 5914 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133774 5914 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133777 5914 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133780 5914 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133784 5914 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133787 5914 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133790 5914 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133793 5914 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133796 5914 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133799 5914 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133802 5914 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133805 5914 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:17:28.139789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133808 5914 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133810 5914 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133814 5914 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133818 5914 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133821 5914 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133825 5914 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133828 5914 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133831 5914 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133834 5914 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133838 5914 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133841 5914 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133843 5914 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133847 5914 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133849 5914 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133854 5914 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133857 5914 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133860 5914 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133863 5914 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133867 5914 flags.go:64] FLAG: --enable-server="true" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133869 5914 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133874 5914 flags.go:64] FLAG: --event-burst="100" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133894 5914 flags.go:64] FLAG: --event-qps="50" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133898 5914 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133901 5914 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:17:28.140341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133904 5914 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133908 5914 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133916 5914 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133919 5914 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133922 5914 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133925 5914 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133928 5914 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133931 5914 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133934 5914 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133937 5914 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133940 5914 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133944 5914 flags.go:64] FLAG: --feature-gates="" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133948 5914 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133951 5914 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133955 5914 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133958 5914 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133962 5914 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133965 5914 flags.go:64] FLAG: --help="false" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133968 5914 flags.go:64] FLAG: --hostname-override="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133971 5914 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133974 5914 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133977 5914 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133980 5914 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:17:28.140926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133983 5914 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133986 5914 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133989 5914 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133991 5914 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133994 5914 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.133997 5914 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134000 5914 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134003 5914 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134007 5914 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134009 5914 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134012 5914 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134017 5914 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134019 5914 flags.go:64] FLAG: --lock-file="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134022 5914 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134025 5914 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134028 5914 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134033 5914 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134036 5914 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134039 5914 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134042 5914 flags.go:64] FLAG: --logging-format="text" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134046 5914 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134050 5914 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134052 5914 flags.go:64] FLAG: --manifest-url="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134055 5914 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134059 5914 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:17:28.141472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134063 5914 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134066 5914 flags.go:64] FLAG: --max-pods="110" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134070 5914 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134072 5914 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134075 5914 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134094 5914 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134097 5914 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134101 5914 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134104 5914 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134111 5914 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134114 5914 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134117 5914 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134121 5914 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134124 5914 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134127 5914 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134130 5914 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134133 5914 flags.go:64] FLAG: --port="10250" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134136 5914 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134141 5914 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a0085c702f29d9d9" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134144 5914 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134147 5914 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134150 5914 flags.go:64] FLAG: --register-node="true" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134153 5914 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134158 5914 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134161 5914 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134163 5914 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:17:28.142111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134166 5914 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134171 5914 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134174 5914 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134177 5914 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134180 5914 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134183 5914 flags.go:64] FLAG: --runonce="false" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134186 5914 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134189 5914 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134192 5914 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134194 5914 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134197 5914 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134200 5914 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134203 5914 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134206 5914 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134209 5914 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134212 5914 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134215 5914 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134217 5914 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134220 5914 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134223 5914 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134226 5914 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134231 5914 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134235 5914 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134237 5914 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134242 5914 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:17:28.142754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134245 5914 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134248 5914 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134251 5914 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134253 5914 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134256 5914 flags.go:64] FLAG: --v="2" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134261 5914 flags.go:64] FLAG: --version="false" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134265 5914 flags.go:64] FLAG: --vmodule="" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134268 5914 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134272 5914 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134403 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134407 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134412 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134415 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134418 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134421 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134424 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134427 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134429 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134432 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134435 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134437 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.143428 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134440 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134443 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134445 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134448 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134450 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134453 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134455 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134458 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134460 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134463 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134468 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134470 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134473 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134475 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134478 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134480 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134483 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134486 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134488 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134493 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.144123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134495 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134498 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134501 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134503 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134507 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134511 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134514 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134516 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134519 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134521 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134524 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134527 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134529 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134532 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134534 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134536 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134539 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134542 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134544 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.144771 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134547 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134549 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134551 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134556 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134559 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134561 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134564 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134566 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134568 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134571 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134574 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134576 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134580 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134582 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134585 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134587 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134590 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134592 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134595 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134597 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.145312 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134600 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134603 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134605 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134608 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134610 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134613 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134615 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134618 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134620 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134623 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134626 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134628 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134631 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134633 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134636 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134640 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134643 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134646 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134648 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134651 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.146160 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134653 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134656 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134658 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134660 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134664 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134667 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134670 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134672 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134675 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134677 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134680 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134682 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134685 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134687 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134690 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.134717 5914 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:17:28.146743 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134751 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134755 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134758 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134760 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134763 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134766 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134768 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134771 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134774 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134776 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134779 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134782 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134785 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134788 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134791 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134794 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134797 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134800 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134802 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134805 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.147238 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134807 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134812 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134814 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134817 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134819 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134822 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134824 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134828 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134830 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134832 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134835 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134838 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134840 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134843 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134845 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134848 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134850 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134853 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134855 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134858 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.147823 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134860 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134863 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134865 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134869 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134872 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134874 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134894 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134897 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134899 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134902 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134904 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134907 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134911 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134916 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134920 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134922 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134925 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134928 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134930 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.148532 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134933 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134936 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134939 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134941 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134944 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134947 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134949 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134952 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134954 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134957 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134960 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134962 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134964 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134967 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134970 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134972 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134976 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134979 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134981 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.149104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134984 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134987 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134990 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134992 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134995 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.134997 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135000 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135004 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135008 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135011 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135014 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135018 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135020 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135023 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135026 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135029 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135032 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135035 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135037 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135040 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.149584 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135042 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135045 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135047 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135050 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135052 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135055 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135057 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.135060 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.139987 5914 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.140003 5914 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.140022 5914 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.140027 5914 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140096 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140102 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140106 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140109 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140112 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140114 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140117 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140120 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.150123 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140123 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140126 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140128 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140131 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140134 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140136 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140139 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140142 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140145 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140148 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140150 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140153 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140156 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140159 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140161 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140165 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140167 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140170 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140172 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140175 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.150634 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140178 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140180 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140183 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140186 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140188 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140191 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140194 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140196 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140199 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140201 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140204 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140208 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140213 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140216 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140220 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140223 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140225 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140228 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140232 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.151191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140234 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140237 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140240 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140242 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140244 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140247 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140250 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140253 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140256 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140258 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140261 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140264 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140266 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140269 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140272 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140275 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140277 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140279 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140282 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140285 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.151683 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140288 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140291 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140293 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140296 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140299 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140301 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140304 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140306 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140308 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140311 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140313 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140316 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140320 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140324 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140327 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140330 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140332 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140335 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140338 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.152338 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140340 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140343 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140346 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140349 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140351 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140354 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140357 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140360 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140362 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140365 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140367 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140370 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140372 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140375 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140377 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140380 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140382 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140385 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140387 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.153095 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140390 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.140417 5914 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140572 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140578 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140581 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140584 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140587 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140590 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140593 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140595 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140598 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140600 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140603 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140606 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140608 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140611 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.153625 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140613 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140616 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140619 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140621 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140624 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140627 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140629 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140632 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140635 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140637 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140640 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140643 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140645 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140648 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140650 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140652 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140655 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140657 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140660 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140662 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.154270 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140665 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140667 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140670 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140673 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140677 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140681 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140684 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140686 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140689 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140692 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140694 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140697 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140699 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140702 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140705 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140708 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140710 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140713 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140715 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140718 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.154925 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140720 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140723 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140726 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140728 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140731 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140733 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140735 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140738 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140740 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140743 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140745 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140748 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140750 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140753 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140755 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140758 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140761 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140764 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140766 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.155602 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140769 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140773 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140777 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140781 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140784 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140786 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140789 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140791 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140794 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140796 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140799 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140801 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140804 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140806 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140809 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140811 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140814 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140816 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.156143 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140819 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140821 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140824 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140826 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140829 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140831 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140833 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140836 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140838 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140841 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140843 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140846 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140848 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140852 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140854 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.140894 5914 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:17:28.156572 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140928 5914 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140933 5914 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140936 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140939 5914 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140942 5914 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140944 5914 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140947 5914 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140950 5914 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140952 5914 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140955 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140958 5914 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140960 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140962 5914 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140965 5914 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140968 5914 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140970 5914 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140973 5914 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140975 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140978 5914 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140981 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:17:28.157025 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140983 5914 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140986 5914 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140988 5914 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140991 5914 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140993 5914 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140996 5914 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.140998 5914 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141001 5914 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141004 5914 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141006 5914 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141009 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141012 5914 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141015 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141017 5914 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141020 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141025 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141027 5914 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141030 5914 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141032 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141035 5914 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:17:28.157550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141038 5914 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141040 5914 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141043 5914 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141045 5914 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141048 5914 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141051 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141054 5914 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141057 5914 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141060 5914 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141064 5914 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141067 5914 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141070 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141072 5914 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141076 5914 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141079 5914 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141081 5914 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141084 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141086 5914 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141089 5914 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141092 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:17:28.158267 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141096 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141098 5914 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141101 5914 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141104 5914 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141106 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141109 5914 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141111 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141116 5914 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141118 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141121 5914 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141123 5914 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141126 5914 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141129 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141131 5914 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141134 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141136 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141139 5914 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141141 5914 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141144 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141147 5914 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:17:28.158750 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141149 5914 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141152 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141154 5914 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141157 5914 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141162 5914 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141166 5914 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141169 5914 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141172 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141175 5914 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141177 5914 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141180 5914 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141183 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141186 5914 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141189 5914 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141192 5914 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141194 5914 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141197 5914 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141200 5914 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141202 5914 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:17:28.159283 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141207 5914 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141209 5914 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141212 5914 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141214 5914 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141217 5914 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141220 5914 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:17:28.141222 5914 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.141276 5914 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.142289 5914 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.142351 5914 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.142522 5914 server.go:1014] "Starting client certificate rotation" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.142658 5914 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.142703 5914 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:11:15 +0000 UTC" deadline="2028-06-19 01:15:52.445856634 +0000 UTC" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.142729 5914 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14331h58m24.303128955s" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.143008 5914 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.144097 5914 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:17:28.159759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.145459 5914 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:17:28.160236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.147530 5914 log.go:25] "Validated CRI v1 image API" Oct 30 21:17:28.160236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.148377 5914 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:17:28.160236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.151191 5914 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c57171df-5c7c-4356-9344-4709e4929e69:/dev/nvme0n1p3 f50dc1a9-7901-418a-8de1-36be5193434b:/dev/nvme0n1p4] Oct 30 21:17:28.160337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.151214 5914 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e/userdata/shm major:0 minor:414 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500/userdata/shm major:0 minor:517 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6/userdata/shm major:0 minor:411 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~projected/kube-api-access-6s7f2:{mountpoint:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~projected/kube-api-access-6s7f2 major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b8f39f3-c5f2-4cde-8a5d-05cd13829521/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0b8f39f3-c5f2-4cde-8a5d-05cd13829521/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~projected/kube-api-access-zmfsk:{mountpoint:/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~projected/kube-api-access-zmfsk major:0 minor:516 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:515 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:510 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12073275-f042-497b-97cc-10cf8d661a68/volumes/kubernetes.io~projected/kube-api-access-8kg9f:{mountpoint:/var/lib/kubelet/pods/12073275-f042-497b-97cc-10cf8d661a68/volumes/kubernetes.io~projected/kube-api-access-8kg9f major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12073275-f042-497b-97cc-10cf8d661a68/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/12073275-f042-497b-97cc-10cf8d661a68/volumes/kubernetes.io~secret/metrics-certs major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b/volumes/kubernetes.io~projected/kube-api-access-hhzhk:{mountpoint:/var/lib/kubelet/pods/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b/volumes/kubernetes.io~projected/kube-api-access-hhzhk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dcf34f-5308-47fd-a7a2-5a277aa95daf/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/21dcf34f-5308-47fd-a7a2-5a277aa95daf/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~projected/kube-api-access-k4mj8:{mountpoint:/var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~projected/kube-api-access-k4mj8 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~projected/kube-api-access-9rxqf:{mountpoint:/var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~projected/kube-api-access-9rxqf major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d658122-115c-4a45-965d-5d75e43ff1be/volumes/kubernetes.io~projected/kube-api-access-csmz2:{mountpoint:/var/lib/kubelet/pods/2d658122-115c-4a45-965d-5d75e43ff1be/volumes/kubernetes.io~projected/kube-api-access-csmz2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/363442c0-a599-49dd-a2d2-a0f93ac8c859/volumes/kubernetes.io~projected/kube-api-access-xfnkq:{mountpoint:/var/lib/kubelet/pods/363442c0-a599-49dd-a2d2-a0f93ac8c859/volumes/kubernetes.io~projected/kube-api-access-xfnkq major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~projected/kube-api-access-9x7tz:{mountpoint:/var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~projected/kube-api-access-9x7tz major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~projected/kube-api-access-xtzg8:{mountpoint:/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~projected/kube-api-access-xtzg8 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volumes/kubernetes.io~projected/kube-api-access-zwdj2:{mountpoint:/var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volumes/kubernetes.io~projected/kube-api-access-zwdj2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd/volumes/kubernetes.io~projected/kube-api-access-bdn5q:{mountpoint:/var/lib/kubelet/pods/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd/volumes/kubernetes.io~projected/kube-api-access-bdn5q major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/506349da-6a0c-482a-a953-3eaa76e552b6/volumes/kubernetes.io~projected/kube-api-access-5rfbd:{mountpoint:/var/lib/kubelet/pods/506349da-6a0c-482a-a953-3eaa76e552b6/volumes/kubernetes.io~projected/kube-api-access-5rfbd major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59caf8ab-703a-4d27-bf2c-faa75cdf4c36/volumes/kubernetes.io~projected/kube-api-access-nfcl5:{mountpoint:/var/lib/kubelet/pods/59caf8ab-703a-4d27-bf2c-faa75cdf4c36/volumes/kubernetes.io~projected/kube-api-access-nfcl5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/720074df-0a6f-4778-bb73-1cd424158dc9/volumes/kubernetes.io~projected/kube-api-access-w95n8:{mountpoint:/var/lib/kubelet/pods/720074df-0a6f-4778-bb73-1cd424158dc9/volumes/kubernetes.io~projected/kube-api-access-w95n8 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/720074df-0a6f-4778-bb73-1cd424158dc9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/720074df-0a6f-4778-bb73-1cd424158dc9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~projected/kube-api-access-tk4dz:{mountpoint:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~projected/kube-api-access-tk4dz major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~projected/kube-api-access-gd62m:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~projected/kube-api-access-gd62m major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~projected/kube-api-access-4dbvb:{mountpoint:/var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~projected/kube-api-access-4dbvb major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/996da354-a8e9-4600-9cad-e8beba46a6ab/volumes/kubernetes.io~projected/kube-api-access-wn7pb:{mountpoint:/var/lib/kubelet/pods/996da354-a8e9-4600-9cad-e8beba46a6ab/volumes/kubernetes.io~projected/kube-api-access-wn7pb major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/996da354-a8e9-4600-9cad-e8beba46a6ab/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/996da354-a8e9-4600-9cad-e8beba46a6ab/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa327463-70c6-4971-9554-b2df6bce21e5/volumes/kubernetes.io~projected/kube-api-access-j4c44:{mountpoint:/var/lib/kubelet/pods/aa327463-70c6-4971-9554-b2df6bce21e5/volumes/kubernetes.io~projected/kube-api-access-j4c44 major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~projected/kube-api-access-twpz4:{mountpoint:/var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~projected/kube-api-access-twpz4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812/volumes/kubernetes.io~projected/kube-api-access-sqg7g:{mountpoint:/var/lib/kubelet/pods/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812/volumes/kubernetes.io~projected/kube-api-access-sqg7g major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2fccb02-6982-45a8-9d67-fe60097a84c3/volumes/kubernetes.io~projected/kube-api-access-n9nd2:{mountpoint:/var/lib/kubelet/pods/e2fccb02-6982-45a8-9d67-fe60097a84c3/volumes/kubernetes.io~projected/kube-api-access-n9nd2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2fccb02-6982-45a8-9d67-fe60097a84c3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e2fccb02-6982-45a8-9d67-fe60097a84c3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/alertmanager/9 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/config-reloader/6 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~empty-dir/config-out major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~projected/kube-api-access-cdrvp:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~projected/kube-api-access-cdrvp major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~projected/tls-assets major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/config-volume major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/web-config major:0 minor:313 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/379e7bfad11bec3858995856122ac69c9804025a9eff7d07bb30c9eb280f37ad/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7b3784c4710d24ff41b9514a5f6fded4d560a322ec685c0991db2038cced0bf7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e3a52c7084a38d13417df6a23577d98e72d74d004c5b2c546f19dcea9c4d5bfc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2e2ab25cb4041d9fdb5f8cecd95b85d1efc59e4e4f2eeec5b6f815819c70729c/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/fb4aaa547be323e976f3a0595d66466626bd69eb1b818a53b93744839170dba2/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e70d4bcb1e3657c96e36d429851ce704af23aa4121ed6512da81c5765150ca36/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8597630d904b4cf0476ec8ff3969d0200a4765f2cead65feef789b3928336184/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5d38c5f9b649740f30283de0909e922cab7c83b0e36d01a03eccb8c28f666b90/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d5ff309aab70b064059fc7de9963e794d25d164565d28abdc4918709481f613e/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/72c38dffb04e6c976fd27a649bc5c55e4794884a7df2d4c6b5646468d05e4254/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4eec772728913c38703e6de4c10a1c97c45c9f7acb746164cda0060232c98da3/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1654ac3206da344540d14df84d796932073b6b3a89a05fa67a1b8e0d2526c1d7/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bbc9683491bbc28310946743f80c704df89649e28a16a1e127cc8c730e846ae6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/c44779ea9d87a62488024d3b7497494b90109ad27a50ff8553bd5a94dfd188fa/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/807f846eff3f160811ec12f99235ceb10a9182767bb66d41e226cc4af7f764c2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e68e95a3f82d24d56e876ce7ea99ab4baf4cad878f4ae6346f53b8ea9a6e936d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bfedf2ff286a06c11fc696f5fa679c87b9e3f9a77388b484b4e47bb5b8f459d0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/136f495cf0518175eff24ddf62a82c41492218250c2a51d971c66d7c2f50c3d4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/eb603aed6e7af392d2ca6ec1bb1a4fe3fbb17bfbfed9cd06db9bd92461c5c76d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/4df605916bfc52ef29f952df2a260c69e6aa48f163ee9bd49a92d7c183cc385c/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/27237e9ce288fa5ed31a0c7d96e91c008b8a48a65b7a2a5badcfbb3b5ee0aeb3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d8877592ffe3611f3bc6c477e2c888c0927a86c529594f1a364f66df07816c68/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e94d322329d5d179f133c83b9239f13adb656df01abd483943b3caec3e46598f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/02a6928f63877995116254bd0e120afdff5b2d30c9e5f139fd8006a3399879cc/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/91559cf286f3a47d5a5d062d7d3adf4bfe2595e3fcf957dcfaf150f67b337d7a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/0a1fcdb92befcd32d73f4991c7dc2809d4324878ea0368d6af04f60039018819/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/a406b71a61ce6a818564cbcc94f38cd29ae99df3a42f38ce61f98338ae41d0a7/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/0518f514824a8573b6aa9d1f94efd9310692de745d0137cb615d9d9f60b81dce/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e249e8e0ebf724c0840b960a46c9669712913a55c5f173dcb2420ec9afc44e56/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/40b736594a2ff417793774fc9202495c5008cdd7078260cddd4bce7a309915c5/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/e99bbd9284b03ca799da31fda2fe8516c84ffd5e75b905925a36a6f2773f8051/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/04c2c1979f2e4eb7cb1856b47894d8532ab30e1c59c731a90873ac31f881136b/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/f1e6a2977b6da1185caa1575d0b0db7ac6641569e33eb37bce621baa8a1eac00/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/4d21e459e454654f5d06307aca1da95a0f9f2f8bb2e01f1c08541cc4ca81c635/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/233b3d384b85b24c17c5458a37d6b7e23daa658e69944b78b0991d58574d9369/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/bfd92ccca10f7f1a9a2e4ab3dd2ba5aa59019a8ac346bea9ab8383314be4bdcc/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/b38eece99fe558d5d15c0b38d73ea456fe389ed471addc32a4a74398661fdadd/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/33c1cb181fec410b4735667e73ce73266eed822f1c599a2f51f1a715adbe93bb/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/90b8be8fa4ab57a62b0585aea08c4e23a11558b6e1b305bb0761b306be4e13cf/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/0a6755a0306fc0ff3b411c499fda6c4740d0af9a49e41e5c98f497ed01034363/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/077310421fdd6059fda1bbf8cb426c8b874b19786ed9ea8812207a27a79c6d7b/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/6ecae7bddea9c90c915ede95c6ad3c88c046e710a31b4903a9ad8dbb7c5d6ad3/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/38342cb6b110b9fe18506da681295d44eb844d4340d818b812fa3a49a6cc33cd/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/2ec0fc77028bf9163aff148d8814b7346c0c26006c9a6a8a62525b8ef6f789b1/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/cabec93c8aaf64cf483e38a86cd063bec22c01757a23bc7cde23f0565f54f94d/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/cc3f6dc45e0eb29c6d71d27c6fe2c9f5c9e9c91810917ea02b5c903e7a0a8ec8/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/178a6d31fe71ca051088afb864124c9c970fe7a1c82e5108078c425a3badc1c9/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/8d1bf3772b9c3abf9e8640aef6239451a64a0e029be65abd7128832d0f8637d3/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/284b698470dfc3e0766078f59fb23ecdfa2ddab23c9dc0636c6e54c0188190d3/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/84c789c3401222107cf2d5ea0ae1df0e868bc04ce5c049089c44e98f2195e30f/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/cc289cbe1ca65c20aaf4e024fba6e9f2cb3aac5f949f4cc8f5af4e83cfe31e69/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/37ecd75a98d7a5086621bfd3f7d0104726af9f5cee9b398c6a0126bf5b1948f9/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/83e5105ba1d828b8a1bab857c9c1a8961da145f9926e9a55758fa4693867c1cb/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/bd46c65566e5e0dfb80af506a9ea72bc0471474403e1c7d3a751cc22be86c141/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/cac719d7cd77cb9a5b9edf609d3aec7d3b3212d36832c3e98ba4f21467e6e2a8/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/f4b2e71c237797f39f265946b985e2c7aa45a41ea4121e37ddf067a70d8541ca/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/b1cb13c987b0311f6f6aeca72726dafb778d9346944b60eb06f45c7f9acf9a0a/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/86c720d60a6e245179d68d568b264901a8b2d37735f04024a8241b84ec628d55/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/edc79b9d751f0db602775b48fc09d5d7807fa553e7def194d6d3f1f9c93c9013/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/b1ee24100f49774530d2f83d85517564abd9a834da4bc59965e0a82d4600a144/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/c91184f141d13d363cade1b1bb52560004828162bace35a1ddbbd3ba22aeef24/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/f0a4ef896bdf2d00e366b9ad638c64208bf736fb9176e4396a8abd45dfec9aca/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/0f4f84c0e22a6fb6074128ccff1b445bfeadf044a57e9dd0a49fb061baba9bcf/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/5752c0ac8442422d83e198418a6ba0dcdef2385fef66d2d7c8df5ef43947e9b5/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/ddb24a60bdd59c3ffc35e74c6bef6b3271b8bc1cf46d1e3622cc5604132c03ea/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/a6c88811ee170ae2871e6c69f853c084a458da944e7db864aaa2a761946ea371/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/59ef36d14014f4fe1e3da51309e1a77e471a5cc885dbbd64ca5dbd7c712820e4/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/09e511b1078bb32d32b1cbd18d7b86dfe6247c7f7497367ff01a21fdf5b96a96/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/673666443ee0108f5d3e9e955b6b0d751a8b89f36712e1fd395f0003cec929ed/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/6e74d67efbd44b324f52f484b0f43e77ec950a79e1f4ed198845ac4ab86b5a6b/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/515ef141af30c162248df28a853b8f31aed93099ee66d70bb2e8f9acbe9176bb/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/726ae7e88e048f5302564b85a3e87775be9d461c28ebf892850d024c99571721/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/e90508bc53cfa445a6eaf91fc66653874fd58da9cbcd47da9956161d6ba0e251/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/9a201ae2310549e8e0f5a500ea8e7e7888b9776bfb1e64424fb2242fb5d85f1e/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/730204d7917ce79159b5a971f07cbdd20cc9ca7850630143def0715fac45e935/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-519:{mountpoint:/var/lib/containers/storage/overlay/feeb13eb9d3423b06ca0178b5ad4a2ded1af83a40c48cc57a21050c83f2bd47f/merged major:0 minor:519 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/fba58e610161280051fe876650c0978ea50e53311289101202db830661202ebb/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/a5adc588e3635da72d6b8221c74055e5cdd721f8eb33d6de6998995c7af5505f/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6e2f38cf1898dfaa1219df5d8cd984e8a8c64a3d41120548063ead70d45f7e1b/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/5d173d403919cfe42fd1677e5bfe09a018dbbfb179f44fc53e05ad0b2ddda307/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/029afd5598e4fcd1f70c81dfd312d5a6a3737328993faf1469245f46c3136ade/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/24ab8e5c24238aa2788e4edd9838cc56632865d0c6ec6762dd821a40e494359f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/4e4764d0596f476f9d45267361ab722121ffdd1ce7c5e0851ff0879f367cd6aa/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/474659934e96f7d6434a794901c92991395df497018ddf24403164b1423c5af7/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/bfe6785bc109df14c6e328beca3a9840098f5df1905e1b0573b548a582f2dbfe/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/2ffa741002fcaaf99ca757076342d1ab086ad0f9d9cb435ef3be74091e201c5d/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9b26869bb57d358f5e863d45d8af6d2d516889aeca63a4cec9d06a1d8595e518/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:17:28.160864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.160358 5914 manager.go:217] Machine: {Timestamp:2026-10-30 21:17:28.159848328 +0000 UTC m=+0.061712384 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100031 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29444563aff6a14108b9dcb408ca7d SystemUUID:ec294445-63af-f6a1-4108-b9dcb408ca7d BootID:69f4b82d-f83a-452b-9c47-09bf2089eacb Filesystems:[{Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~projected/kube-api-access-9x7tz DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:385 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36f6ae48d1e7ceadaf059b42c092e4eddeb369a53334cd6e2eb9167248a29718/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aa327463-70c6-4971-9554-b2df6bce21e5/volumes/kubernetes.io~projected/kube-api-access-j4c44 DeviceMajor:0 DeviceMinor:410 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2d658122-115c-4a45-965d-5d75e43ff1be/volumes/kubernetes.io~projected/kube-api-access-csmz2 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/720074df-0a6f-4778-bb73-1cd424158dc9/volumes/kubernetes.io~projected/kube-api-access-w95n8 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/363442c0-a599-49dd-a2d2-a0f93ac8c859/volumes/kubernetes.io~projected/kube-api-access-xfnkq DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volumes/kubernetes.io~projected/kube-api-access-zwdj2 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dd7a1f9e2c40526a94c5f189ba2ff709a6cf65e56a0ba3902611faab3c60df1/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5eaf662caf21d49cc50c4f929cb5cc1f63bed77a0df65f1edf33c9d0f75e7621/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2fccb02-6982-45a8-9d67-fe60097a84c3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:374 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21dcf34f-5308-47fd-a7a2-5a277aa95daf/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:404 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2cb3de8cb4f1fd8700c36ace4488f4233dd36af2620717ac84a02ef422399b11/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59caf8ab-703a-4d27-bf2c-faa75cdf4c36/volumes/kubernetes.io~projected/kube-api-access-nfcl5 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afd3d5eb770d5a36764d57d0eb798b0246b30c9e3bc48eea1733c17fa8d0275a/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:355 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36296f5ff6a2c42ca75e06302da2f4149c94683e457d9a0f791b0fcab8a36b3b/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e8d25606b10cf4824552fc616962b863ef5ccaddf2970f005791ca52cac5768/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:309 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9c71991bb1186e86f6c7e334a752e1cd4d3a1d68b2ac4203ec5f2dc73e2e2a6/userdata/shm DeviceMajor:0 DeviceMinor:411 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c370fc89afff44194e342f9cac351a69620b821ceb887bbc001a3d92cd3f77f/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a535a3c3b2e4d446190af727eeb1adb2801447d4293b3bd08cbb8ee133cc40d/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:373 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e92fddc1f18bbd7eda580c04bc3c854314658f9152fee48e7bf4c76f390761d/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:510 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-519 DeviceMajor:0 DeviceMinor:519 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2fccb02-6982-45a8-9d67-fe60097a84c3/volumes/kubernetes.io~projected/kube-api-access-n9nd2 DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/355b83170276f1fe114d37cc37a073d7525a2ca6e61d7a35b7176d1347fa99f4/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:275 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~projected/kube-api-access-zmfsk DeviceMajor:0 DeviceMinor:516 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:384 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e763e1d706d122813d3f0a274968b8287dda890d82e3781920ddc7a08789f7e/userdata/shm DeviceMajor:0 DeviceMinor:414 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:353 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/996da354-a8e9-4600-9cad-e8beba46a6ab/volumes/kubernetes.io~projected/kube-api-access-wn7pb DeviceMajor:0 DeviceMinor:366 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12073275-f042-497b-97cc-10cf8d661a68/volumes/kubernetes.io~projected/kube-api-access-8kg9f DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~projected/kube-api-access-xtzg8 DeviceMajor:0 DeviceMinor:301 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:386 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~projected/kube-api-access-cdrvp DeviceMajor:0 DeviceMinor:321 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~projected/kube-api-access-6s7f2 DeviceMajor:0 DeviceMinor:387 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6e904f9b62cb78aea134ae1876293956e9aaf7e6cb1ab3488a2b91e3d24154f/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/720074df-0a6f-4778-bb73-1cd424158dc9/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4245b5a9bac428e54ac809f86b631c7e1f7281ce8395002c28336a5bfd208f1/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:354 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0b8f39f3-c5f2-4cde-8a5d-05cd13829521/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13134147995eb5fd5095acb838c9b54d5c135ce18c89ff6ef4d800bc1ae24b33/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/985581c6-d015-4770-992d-b75ab7f0654a/volumes/kubernetes.io~projected/kube-api-access-4dbvb DeviceMajor:0 DeviceMinor:273 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58150129d11ce7f13181d106546461c5e45eaee03144346428f253e94ec80b1c/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18528bd2d9b0c1c1ea70ef61361d2a26becf0d4c00beb1ba9dc7e5d8715f54f3/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/506349da-6a0c-482a-a953-3eaa76e552b6/volumes/kubernetes.io~projected/kube-api-access-5rfbd DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3877d90b-6949-48e6-bdd5-568a248a31d9/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:318 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~projected/kube-api-access-gd62m DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/996da354-a8e9-4600-9cad-e8beba46a6ab/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:363 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:515 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4229762d-c3ea-43fd-851d-506bba6c76db/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e39a5665e12f91299e96814409dd1aab3d8e8c3603fc3f0b6641a25d44d3ff3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd/volumes/kubernetes.io~projected/kube-api-access-bdn5q DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bfc67d0-5890-47b5-9885-4338121de31e/volumes/kubernetes.io~projected/kube-api-access-k4mj8 DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db5959c6a8990cd5956941df5c6a5a642b8a743504a2c4307121c1f1572f8508/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:342 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12073275-f042-497b-97cc-10cf8d661a68/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:399 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b/volumes/kubernetes.io~projected/kube-api-access-hhzhk DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b29ee7c6-efd8-4404-8a4d-de677037375e/volumes/kubernetes.io~projected/kube-api-access-twpz4 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1c6a982f85989d08d92e39b464483568a9af6e393ad6a4d0e19b9d4e6b9edaa/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83d1f5dcecc7cd18b070d2979154567f53587bdf467810e5e49b71ee62035daa/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe/volumes/kubernetes.io~projected/kube-api-access-9rxqf DeviceMajor:0 DeviceMinor:274 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e7aad03-570e-43d0-bde7-83dd1740ac78/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:343 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812/volumes/kubernetes.io~projected/kube-api-access-sqg7g DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a04c0a7-b90d-4233-bc46-b8430095ef21/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:383 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500/userdata/shm DeviceMajor:0 DeviceMinor:517 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7496f3c4-d42d-446d-8c0e-32272aa84f34/volumes/kubernetes.io~projected/kube-api-access-tk4dz DeviceMajor:0 DeviceMinor:375 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e763e1d706d122 MacAddress:b6:da:a7:d1:80:b2 Speed:10000 Mtu:8901} {Name:13134147995eb5f MacAddress:aa:0d:ab:00:3f:45 Speed:10000 Mtu:8901} {Name:2a535a3c3b2e4d4 MacAddress:56:29:7e:e9:df:27 Speed:10000 Mtu:8901} {Name:2c370fc89afff44 MacAddress:f2:c3:7e:5e:2a:ea Speed:10000 Mtu:8901} {Name:36296f5ff6a2c42 MacAddress:a6:20:31:65:af:68 Speed:10000 Mtu:8901} {Name:36f6ae48d1e7cea MacAddress:96:f2:cc:3b:49:78 Speed:10000 Mtu:8901} {Name:5a169ce2bbb9ddb MacAddress:96:12:4e:60:a5:ad Speed:10000 Mtu:8901} {Name:5eaf662caf21d49 MacAddress:6a:2e:d0:5c:91:43 Speed:10000 Mtu:8901} {Name:64fff64eb4eb407 MacAddress:6e:50:7e:60:4a:55 Speed:10000 Mtu:8901} {Name:6e8d25606b10cf4 MacAddress:66:49:46:73:7f:3e Speed:10000 Mtu:8901} {Name:7e92fddc1f18bbd MacAddress:fe:d6:3c:8c:4d:a6 Speed:10000 Mtu:8901} {Name:83d1f5dcecc7cd1 MacAddress:ea:ab:e4:0d:84:e0 Speed:10000 Mtu:8901} {Name:afd3d5eb770d5a3 MacAddress:26:2f:6b:5a:42:33 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:75:37:e3:5c:73 Speed:0 Mtu:9001} {Name:br-int MacAddress:96:0d:8d:4d:eb:8a Speed:0 Mtu:8901} {Name:d9c71991bb1186e MacAddress:ee:cc:58:45:da:38 Speed:10000 Mtu:8901} {Name:db5959c6a8990cd MacAddress:f2:85:e4:a9:4f:3b Speed:10000 Mtu:8901} {Name:ec977eb79a06132 MacAddress:d2:d6:a7:d4:b0:28 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:75:37:e3:5c:73 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ca:51:65:05:73:45 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:ba:06:cc:8e:d9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:17:28.161250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.160868 5914 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:17:28.161250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.160936 5914 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:17:28.161356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161299 5914 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:17:28.161519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161341 5914 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-119.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:17:28.161596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161522 5914 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:17:28.161596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161535 5914 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:17:28.161596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161549 5914 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:17:28.161596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161569 5914 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:17:28.161751 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161709 5914 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:17:28.161952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.161938 5914 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:17:28.162094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162082 5914 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:17:28.162149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162133 5914 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:17:28.162149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162147 5914 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:17:28.162233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162164 5914 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:17:28.162233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162178 5914 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:17:28.162233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162189 5914 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:17:28.162912 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162896 5914 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:17:28.162912 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.162911 5914 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:17:28.163480 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.163459 5914 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:17:28.166589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.166574 5914 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:17:28.166589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.166589 5914 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:17:28.166698 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.166688 5914 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:17:28.166908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.166896 5914 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:17:28.166942 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.166935 5914 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:17:28.167067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167055 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167076 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167085 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167092 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167100 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167106 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167112 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:17:28.167115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167117 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:17:28.167377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167124 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:17:28.167377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167130 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:17:28.167377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167142 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:17:28.167377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167151 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:17:28.167377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167178 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:17:28.167377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.167184 5914 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:17:28.170595 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.170582 5914 server.go:1263] "Started kubelet" Oct 30 21:17:28.170784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.170676 5914 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:17:28.170850 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.170811 5914 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:17:28.171101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.170997 5914 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:17:28.171180 ip-10-0-142-119 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:17:28.171732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.171593 5914 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:17:28.175371 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.172870 5914 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:17:28.178093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.178072 5914 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:17:28.178207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.178155 5914 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:17:28.178387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.178360 5914 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:11:17 +0000 UTC" deadline="2028-06-01 15:33:15.360771947 +0000 UTC" Oct 30 21:17:28.178442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.178389 5914 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13914h15m47.182386298s" Oct 30 21:17:28.178442 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.178425 5914 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:17:28.178771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.178740 5914 factory.go:55] Registering systemd factory Oct 30 21:17:28.178771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.178764 5914 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:17:28.179222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179204 5914 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:17:28.179222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179222 5914 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:17:28.179358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179293 5914 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:17:28.182175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179480 5914 factory.go:153] Registering CRI-O factory Oct 30 21:17:28.182175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179496 5914 factory.go:223] Registration of the crio container factory successfully Oct 30 21:17:28.182175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179545 5914 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:17:28.182175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179562 5914 factory.go:103] Registering Raw factory Oct 30 21:17:28.182175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179574 5914 manager.go:1199] Started watching for new ooms in manager Oct 30 21:17:28.182175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.179829 5914 manager.go:319] Starting recovery of all containers Oct 30 21:17:28.182751 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182710 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/projected/0a04c0a7-b90d-4233-bc46-b8430095ef21-kube-api-access-6s7f2" seLinuxMountContext="" Oct 30 21:17:28.182817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182764 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 21:17:28.182817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182779 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996da354-a8e9-4600-9cad-e8beba46a6ab" volumeName="kubernetes.io/secret/996da354-a8e9-4600-9cad-e8beba46a6ab-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 21:17:28.182817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182791 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2fccb02-6982-45a8-9d67-fe60097a84c3" volumeName="kubernetes.io/secret/e2fccb02-6982-45a8-9d67-fe60097a84c3-metrics-tls" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182825 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182859 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182873 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-client-tls" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182914 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182929 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182956 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb13caf-5f94-4a1a-a865-8f3a1ee584fe" volumeName="kubernetes.io/empty-dir/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-volume-directive-shadow" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182978 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2fccb02-6982-45a8-9d67-fe60097a84c3" volumeName="kubernetes.io/projected/e2fccb02-6982-45a8-9d67-fe60097a84c3-kube-api-access-n9nd2" seLinuxMountContext="" Oct 30 21:17:28.182996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.182990 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183002 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183023 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bfc67d0-5890-47b5-9885-4338121de31e" volumeName="kubernetes.io/projected/2bfc67d0-5890-47b5-9885-4338121de31e-kube-api-access-k4mj8" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183040 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d658122-115c-4a45-965d-5d75e43ff1be" volumeName="kubernetes.io/configmap/2d658122-115c-4a45-965d-5d75e43ff1be-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183080 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d658122-115c-4a45-965d-5d75e43ff1be" volumeName="kubernetes.io/projected/2d658122-115c-4a45-965d-5d75e43ff1be-kube-api-access-csmz2" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183100 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3877d90b-6949-48e6-bdd5-568a248a31d9" volumeName="kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183127 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183138 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4229762d-c3ea-43fd-851d-506bba6c76db" volumeName="kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183159 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4229762d-c3ea-43fd-851d-506bba6c76db" volumeName="kubernetes.io/secret/4229762d-c3ea-43fd-851d-506bba6c76db-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183173 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="985581c6-d015-4770-992d-b75ab7f0654a" volumeName="kubernetes.io/empty-dir/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183184 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd" volumeName="kubernetes.io/projected/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-kube-api-access-bdn5q" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183204 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183216 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183232 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b8f39f3-c5f2-4cde-8a5d-05cd13829521" volumeName="kubernetes.io/configmap/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-konnectivity-ca" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183261 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183291 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363442c0-a599-49dd-a2d2-a0f93ac8c859" volumeName="kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183314 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720074df-0a6f-4778-bb73-1cd424158dc9" volumeName="kubernetes.io/secret/720074df-0a6f-4778-bb73-1cd424158dc9-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183338 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183353 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3877d90b-6949-48e6-bdd5-568a248a31d9" volumeName="kubernetes.io/projected/3877d90b-6949-48e6-bdd5-568a248a31d9-kube-api-access-9x7tz" seLinuxMountContext="" Oct 30 21:17:28.183372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183364 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183385 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12073275-f042-497b-97cc-10cf8d661a68" volumeName="kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183397 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bfc67d0-5890-47b5-9885-4338121de31e" volumeName="kubernetes.io/configmap/2bfc67d0-5890-47b5-9885-4338121de31e-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183423 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363442c0-a599-49dd-a2d2-a0f93ac8c859" volumeName="kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183443 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183455 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a1b8a3-1ae3-4724-ba2e-7260a79c6812" volumeName="kubernetes.io/projected/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-kube-api-access-sqg7g" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183475 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0fea2c-ba27-4a1c-a48d-c070d9577b1b" volumeName="kubernetes.io/empty-dir/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-tmp-dir" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183496 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bfc67d0-5890-47b5-9885-4338121de31e" volumeName="kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-tls" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183524 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4229762d-c3ea-43fd-851d-506bba6c76db" volumeName="kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-env-overrides" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183536 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506349da-6a0c-482a-a953-3eaa76e552b6" volumeName="kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-multus-daemon-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183557 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="985581c6-d015-4770-992d-b75ab7f0654a" volumeName="kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183575 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="985581c6-d015-4770-992d-b75ab7f0654a" volumeName="kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183587 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b29ee7c6-efd8-4404-8a4d-de677037375e" volumeName="kubernetes.io/projected/b29ee7c6-efd8-4404-8a4d-de677037375e-kube-api-access-twpz4" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183622 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183637 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b8f39f3-c5f2-4cde-8a5d-05cd13829521" volumeName="kubernetes.io/secret/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-agent-certs" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183650 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/configmap/7e7aad03-570e-43d0-bde7-83dd1740ac78-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183670 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183681 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183693 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363442c0-a599-49dd-a2d2-a0f93ac8c859" volumeName="kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-binary-copy" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183725 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720074df-0a6f-4778-bb73-1cd424158dc9" volumeName="kubernetes.io/empty-dir/720074df-0a6f-4778-bb73-1cd424158dc9-data-volume" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183738 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183748 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="985581c6-d015-4770-992d-b75ab7f0654a" volumeName="kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-tls" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183779 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b29ee7c6-efd8-4404-8a4d-de677037375e" volumeName="kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-tuned" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183791 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a1b8a3-1ae3-4724-ba2e-7260a79c6812" volumeName="kubernetes.io/configmap/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-serviceca" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183819 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2fccb02-6982-45a8-9d67-fe60097a84c3" volumeName="kubernetes.io/configmap/e2fccb02-6982-45a8-9d67-fe60097a84c3-config-volume" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183870 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb13caf-5f94-4a1a-a865-8f3a1ee584fe" volumeName="kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183938 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb13caf-5f94-4a1a-a865-8f3a1ee584fe" volumeName="kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.183979 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3877d90b-6949-48e6-bdd5-568a248a31d9" volumeName="kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184000 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506349da-6a0c-482a-a953-3eaa76e552b6" volumeName="kubernetes.io/projected/506349da-6a0c-482a-a953-3eaa76e552b6-kube-api-access-5rfbd" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184012 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b29ee7c6-efd8-4404-8a4d-de677037375e" volumeName="kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-tmp" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184024 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184047 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184058 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59caf8ab-703a-4d27-bf2c-faa75cdf4c36" volumeName="kubernetes.io/projected/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kube-api-access-nfcl5" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184069 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184091 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184105 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd" volumeName="kubernetes.io/secret/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-cert" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184116 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184128 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/projected/7496f3c4-d42d-446d-8c0e-32272aa84f34-kube-api-access-tk4dz" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184143 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184172 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184183 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="985581c6-d015-4770-992d-b75ab7f0654a" volumeName="kubernetes.io/projected/985581c6-d015-4770-992d-b75ab7f0654a-kube-api-access-4dbvb" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184197 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="985581c6-d015-4770-992d-b75ab7f0654a" volumeName="kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184208 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="363442c0-a599-49dd-a2d2-a0f93ac8c859" volumeName="kubernetes.io/projected/363442c0-a599-49dd-a2d2-a0f93ac8c859-kube-api-access-xfnkq" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184220 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184232 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-federate-client-tls" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184245 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184256 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184268 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f0fea2c-ba27-4a1c-a48d-c070d9577b1b" volumeName="kubernetes.io/projected/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-kube-api-access-hhzhk" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184279 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb13caf-5f94-4a1a-a865-8f3a1ee584fe" volumeName="kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184292 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="506349da-6a0c-482a-a953-3eaa76e552b6" volumeName="kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-cni-binary-copy" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184303 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert" seLinuxMountContext="" Oct 30 21:17:28.184293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184316 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184328 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184343 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184367 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720074df-0a6f-4778-bb73-1cd424158dc9" volumeName="kubernetes.io/projected/720074df-0a6f-4778-bb73-1cd424158dc9-kube-api-access-w95n8" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184379 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/empty-dir/7496f3c4-d42d-446d-8c0e-32272aa84f34-audit-log" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184400 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184413 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2fccb02-6982-45a8-9d67-fe60097a84c3" volumeName="kubernetes.io/empty-dir/e2fccb02-6982-45a8-9d67-fe60097a84c3-tmp-dir" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184434 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bfc67d0-5890-47b5-9885-4338121de31e" volumeName="kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184446 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3877d90b-6949-48e6-bdd5-568a248a31d9" volumeName="kubernetes.io/configmap/3877d90b-6949-48e6-bdd5-568a248a31d9-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184457 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/empty-dir/7e7aad03-570e-43d0-bde7-83dd1740ac78-tmp" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184469 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184481 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="996da354-a8e9-4600-9cad-e8beba46a6ab" volumeName="kubernetes.io/projected/996da354-a8e9-4600-9cad-e8beba46a6ab-kube-api-access-wn7pb" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184496 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184516 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184530 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184541 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a04c0a7-b90d-4233-bc46-b8430095ef21" volumeName="kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184555 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12073275-f042-497b-97cc-10cf8d661a68" volumeName="kubernetes.io/projected/12073275-f042-497b-97cc-10cf8d661a68-kube-api-access-8kg9f" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184566 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dcf34f-5308-47fd-a7a2-5a277aa95daf" volumeName="kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184580 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb13caf-5f94-4a1a-a865-8f3a1ee584fe" volumeName="kubernetes.io/projected/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-api-access-9rxqf" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184592 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184612 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7496f3c4-d42d-446d-8c0e-32272aa84f34" volumeName="kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-client-ca-bundle" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184624 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e7aad03-570e-43d0-bde7-83dd1740ac78" volumeName="kubernetes.io/projected/7e7aad03-570e-43d0-bde7-83dd1740ac78-kube-api-access-gd62m" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184639 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" volumeName="kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184653 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" volumeName="kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184667 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb13caf-5f94-4a1a-a865-8f3a1ee584fe" volumeName="kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184679 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184690 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39ab801c-ab61-43bb-8523-8798c89b5279" volumeName="kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184702 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4229762d-c3ea-43fd-851d-506bba6c76db" volumeName="kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-config" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184717 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4229762d-c3ea-43fd-851d-506bba6c76db" volumeName="kubernetes.io/projected/4229762d-c3ea-43fd-851d-506bba6c76db-kube-api-access-zwdj2" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184732 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720074df-0a6f-4778-bb73-1cd424158dc9" volumeName="kubernetes.io/configmap/720074df-0a6f-4778-bb73-1cd424158dc9-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184743 5914 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa327463-70c6-4971-9554-b2df6bce21e5" volumeName="kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44" seLinuxMountContext="" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184760 5914 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:17:28.186057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.184767 5914 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:17:28.207130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.207090 5914 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:17:28.208810 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.208785 5914 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:17:28.208810 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.208811 5914 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:17:28.208990 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.208842 5914 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:17:28.208990 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.208902 5914 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:17:28.227729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.227692 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:17:28.227865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.227696 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:17:28.234145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.234110 5914 generic.go:356] "Generic (PLEG): container finished" podID="4a4c6d46aa36b1eea7dc39743f5eb2b7" containerID="2b45e00b3eb9e91d8cc22770d9e2e9e506523e14511e68928d88bf2dd0a9a0e1" exitCode=0 Oct 30 21:17:28.239851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.239820 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" exitCode=0 Oct 30 21:17:28.245776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.245755 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:17:28.246151 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.246128 5914 generic.go:356] "Generic (PLEG): container finished" podID="4229762d-c3ea-43fd-851d-506bba6c76db" containerID="e8ae4035fa0ad1e575f66d36ba1f0049b00f7f0e627356bfd394edec9d1dfd5b" exitCode=1 Oct 30 21:17:28.250420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.250393 5914 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="6d3f1dc8ec9b1c8549ad9a786b42338368bcd184b44e48b05eb70838d1407946" exitCode=0 Oct 30 21:17:28.250420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.250420 5914 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="1248e4f320cd33795eccdb1425f6193de2125cc316d38ef1bfc04b76719b4438" exitCode=0 Oct 30 21:17:28.250557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.250434 5914 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="3fea154c923779c3eac4a2e3a0a05f1b04be65fbc806c3cebef0211632161d36" exitCode=0 Oct 30 21:17:28.250557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.250443 5914 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="f2e1606b7b4cfaefdf2af00e08b661ea2c170528846df5354a8e7324a9be04e7" exitCode=0 Oct 30 21:17:28.250557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.250451 5914 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="db6c816284df1924a75024b182ffd66f64b7ec08b3d3fe88142ee2b3b4f8e2ef" exitCode=0 Oct 30 21:17:28.250557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.250468 5914 generic.go:356] "Generic (PLEG): container finished" podID="363442c0-a599-49dd-a2d2-a0f93ac8c859" containerID="7f9752885c9f7c7b9c70c0bfe900708f68829dc75623886041656213d87b9a23" exitCode=0 Oct 30 21:17:28.278497 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.278468 5914 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-119.us-west-1.compute.internal\" not found" Oct 30 21:17:28.284446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.284411 5914 generic.go:356] "Generic (PLEG): container finished" podID="985581c6-d015-4770-992d-b75ab7f0654a" containerID="b52bf282cf583348698dcc5487aaf6e2b272e468b2a21446e5661ed863308f4c" exitCode=0 Oct 30 21:17:28.309056 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.308981 5914 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:17:28.328391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.328368 5914 manager.go:324] Recovery completed Oct 30 21:17:28.357305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.357272 5914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:17:28.359441 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.359407 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:17:28.359441 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.359440 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:17:28.359635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.359450 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:17:28.364705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.364684 5914 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:17:28.364705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.364702 5914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:17:28.364923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.364702 5914 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:17:28.364923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.364816 5914 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:17:28.365031 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365013 5914 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:17:28.365070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365035 5914 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:17:28.365070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365060 5914 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:17:28.365124 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365069 5914 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:17:28.365124 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365078 5914 policy_none.go:50] "Start" Oct 30 21:17:28.365124 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365093 5914 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:17:28.365124 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365106 5914 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:17:28.365310 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365297 5914 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:17:28.365341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365313 5914 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:17:28.365341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365321 5914 policy_none.go:44] "Start" Oct 30 21:17:28.365831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365816 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:17:28.365928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365846 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:17:28.365928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.365858 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:17:28.371756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.371734 5914 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:17:28.371868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.371818 5914 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:17:28.371868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.371836 5914 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:17:28.371957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.371937 5914 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:17:28.372187 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.372163 5914 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:17:28.376732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.376717 5914 manager.go:341] "Starting Device Plugin manager" Oct 30 21:17:28.377248 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.376749 5914 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:17:28.377248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377253 5914 server.go:87] "Starting device plugin registration server" Oct 30 21:17:28.377599 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377584 5914 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:17:28.377698 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377600 5914 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:17:28.377817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377803 5914 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:17:28.377959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377930 5914 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:17:28.377999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377968 5914 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:17:28.377999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.377976 5914 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:17:28.378111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.378046 5914 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:17:28.377958748Z","UUID":"5f3d7c0f-b030-41fe-96f5-1d533d41494b","Handler":null,"Name":"","Endpoint":""} Oct 30 21:17:28.380322 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.380012 5914 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:17:28.380322 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.380039 5914 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:17:28.380587 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.380568 5914 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:17:28.469570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.469535 5914 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:17:28.478102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.478075 5914 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:17:28.479240 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.479223 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:17:28.479292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.479259 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:17:28.479292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.479273 5914 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-119.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:17:28.479373 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.479323 5914 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.509581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.509511 5914 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal"] Oct 30 21:17:28.510405 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510382 5914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.510644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510425 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"d6458e4c6fa120c378f995c71dc796f0","Type":"ContainerStarted","Data":"3d382d92caded67191c88364acab0a437f6e6ae38784b585e28e44a123b35898"} Oct 30 21:17:28.510644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510472 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"d6458e4c6fa120c378f995c71dc796f0","Type":"ContainerStarted","Data":"4a842b88448abd8503aba225d6c2e936f46a95cec73f264c251f0689aa982a5d"} Oct 30 21:17:28.510644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510512 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"4a4c6d46aa36b1eea7dc39743f5eb2b7","Type":"ContainerStarted","Data":"72da923226da103f5c9c1effee3b81fe45f8992c3f12832e1d83fda3b3989f85"} Oct 30 21:17:28.510644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510531 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"4a4c6d46aa36b1eea7dc39743f5eb2b7","Type":"ContainerDied","Data":"2b45e00b3eb9e91d8cc22770d9e2e9e506523e14511e68928d88bf2dd0a9a0e1"} Oct 30 21:17:28.510644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510549 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" event={"ID":"4a4c6d46aa36b1eea7dc39743f5eb2b7","Type":"ContainerStarted","Data":"94136c348d5090aa84a92f15988d15601268259dc8f3ded19fc5eef79ccd6715"} Oct 30 21:17:28.510950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.510896 5914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.572614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.572540 5914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:17:28.572614 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.572604 5914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.572872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.572778 5914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:17:28.572872 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:28.572830 5914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.589213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.589186 5914 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.589346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.589262 5914 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.673621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.673593 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.673772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.673627 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.673772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.673650 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6458e4c6fa120c378f995c71dc796f0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"d6458e4c6fa120c378f995c71dc796f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.774432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.774396 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.774432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.774347 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.774612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.774471 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6458e4c6fa120c378f995c71dc796f0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"d6458e4c6fa120c378f995c71dc796f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.774612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.774494 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.774612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.774523 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4a4c6d46aa36b1eea7dc39743f5eb2b7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"4a4c6d46aa36b1eea7dc39743f5eb2b7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:28.774612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:28.774556 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6458e4c6fa120c378f995c71dc796f0-config\") pod \"kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal\" (UID: \"d6458e4c6fa120c378f995c71dc796f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:29.163154 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.163120 5914 apiserver.go:52] "Watching apiserver" Oct 30 21:17:29.295594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.295566 5914 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:29.299146 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.299127 5914 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:17:29.299994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.299974 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-5bz2b","openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f","openshift-multus/multus-t6fc6","openshift-multus/network-metrics-daemon-kzs5d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c","openshift-monitoring/metrics-server-6dcc758dc7-djcnn","openshift-console/console-75748b998b-p9kst","openshift-insights/insights-runtime-extractor-7bckf","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4","openshift-network-operator/iptables-alerter-5m86j","kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-ndqrr","openshift-multus/multus-additional-cni-plugins-d2f8g","openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll","openshift-ovn-kubernetes/ovnkube-node-69lqx","openshift-monitoring/node-exporter-qz9gs","openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z","openshift-cluster-node-tuning-operator/tuned-x6zm6","openshift-dns/dns-default-qx48k","openshift-console/console-85894779cf-c48vt","openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b","kube-system/global-pull-secret-syncer-5vztw","kube-system/konnectivity-agent-9q452","openshift-dns/node-resolver-67mbf","openshift-monitoring/telemeter-client-8dcd9788c-w4vk7","openshift-network-diagnostics/network-check-target-h8c58"] Oct 30 21:17:29.355533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355477 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-p2m87\"" Oct 30 21:17:29.355533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355496 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:17:29.355533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355516 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-msgtv\"" Oct 30 21:17:29.355533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355544 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrcct\"" Oct 30 21:17:29.356010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355677 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:17:29.356010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355682 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lps2\"" Oct 30 21:17:29.356010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355707 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:17:29.356010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.355961 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.356166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356008 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:17:29.356166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356030 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fp6d9\"" Oct 30 21:17:29.356166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356034 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.356289 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356275 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.356348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356308 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ks46b\"" Oct 30 21:17:29.356348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356321 5914 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:17:29.356348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356334 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356378 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356383 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356380 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356387 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:29.356427 5914 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-119.us-west-1.compute.internal" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356450 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356454 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.356533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356536 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356551 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356591 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356659 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ztz2s\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356659 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356683 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356683 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356746 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:17:29.356876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.356799 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p6llv\"" Oct 30 21:17:29.382307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.382284 5914 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:17:29.401350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401321 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:17:29.401448 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401380 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.401633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401605 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.401633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401608 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.401744 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401682 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.401788 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401759 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.401847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401796 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.401847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401837 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:17:29.401972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.401865 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.402181 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402156 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:17:29.402181 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402159 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:17:29.402327 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402192 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:17:29.402539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402520 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.402619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402547 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:17:29.402619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402562 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:17:29.402619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402576 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fgt7b\"" Oct 30 21:17:29.402619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402583 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:17:29.402787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402765 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:17:29.402787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402773 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:17:29.402900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402773 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wvjk\"" Oct 30 21:17:29.402900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402858 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:17:29.402900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402867 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:17:29.402900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.402868 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:17:29.403333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403316 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:17:29.403429 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403344 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:17:29.403429 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403425 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42q2c\"" Oct 30 21:17:29.403573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403375 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:17:29.403573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403456 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:17:29.403573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403502 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.403573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403509 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-crhr9\"" Oct 30 21:17:29.403737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403647 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.403936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403915 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:17:29.403936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.403916 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8tbzb\"" Oct 30 21:17:29.404082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404032 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:17:29.404082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404058 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:17:29.404082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404076 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:17:29.404196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404191 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:17:29.404329 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404315 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 21:17:29.404402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404327 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 21:17:29.404402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404370 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:17:29.404483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404403 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:17:29.404483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404406 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:17:29.404579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404492 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:17:29.404579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404511 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:17:29.404579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404530 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:17:29.404579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404516 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:17:29.404579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404530 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:17:29.404579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404564 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404592 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404608 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qn5sj\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404513 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404609 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404630 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404550 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404570 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404643 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:17:29.404779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404694 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:17:29.405029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.404973 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:17:29.405392 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.405380 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 21:17:29.405499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.405489 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:17:29.405539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.405521 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-b4tkk\"" Oct 30 21:17:29.405603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.405586 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:17:29.405950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.405930 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-84xd8\"" Oct 30 21:17:29.406100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.406082 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8g8v4\"" Oct 30 21:17:29.406357 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.406337 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r4rjf\"" Oct 30 21:17:29.406553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.406526 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kknvl\"" Oct 30 21:17:29.406771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.406748 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:17:29.407104 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.407086 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9jlcd\"" Oct 30 21:17:29.407218 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.407205 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmnhg\"" Oct 30 21:17:29.407374 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.407348 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a115kei9rjd2d\"" Oct 30 21:17:29.407984 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.407960 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-57txw\"" Oct 30 21:17:29.408080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.408030 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:17:29.408186 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.408174 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nxjfg\"" Oct 30 21:17:29.408319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.408306 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3l2gij8kn99ct\"" Oct 30 21:17:29.408870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.408854 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5jmhs\"" Oct 30 21:17:29.408964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.408923 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 21:17:29.408964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.408958 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:17:29.409095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.409046 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:17:29.409160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.409113 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z7dj2\"" Oct 30 21:17:29.409160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.409148 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 21:17:29.409265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.409196 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:17:29.453025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.452874 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:17:29.453815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.453795 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:17:29.453919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.453844 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 21:17:29.478648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478624 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.478759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478651 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpz4\" (UniqueName: \"kubernetes.io/projected/b29ee7c6-efd8-4404-8a4d-de677037375e-kube-api-access-twpz4\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.478759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478670 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.478759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478685 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nd2\" (UniqueName: \"kubernetes.io/projected/e2fccb02-6982-45a8-9d67-fe60097a84c3-kube-api-access-n9nd2\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.478759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478710 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-cnibin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.478759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478732 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-bin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.478759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478754 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478901 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-modprobe-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478942 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcl5\" (UniqueName: \"kubernetes.io/projected/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kube-api-access-nfcl5\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.478970 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7tz\" (UniqueName: \"kubernetes.io/projected/3877d90b-6949-48e6-bdd5-568a248a31d9-kube-api-access-9x7tz\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479008 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479013 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479026 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-run\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.479047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479051 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-sys\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.479316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479106 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.479316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479150 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-tls\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.479316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479260 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.479316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479298 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.479316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479304 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-tls\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.479467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479405 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.479467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479428 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-config\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.479467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479455 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-host\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479471 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-tmp\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479488 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-textfile\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479526 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479562 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-textfile\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479567 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbvb\" (UniqueName: \"kubernetes.io/projected/985581c6-d015-4770-992d-b75ab7f0654a-kube-api-access-4dbvb\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479580 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-tmp\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.479596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479594 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-cni-binary-copy\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479699 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-config\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479700 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/720074df-0a6f-4778-bb73-1cd424158dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479746 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479753 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479758 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-cni-binary-copy\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479777 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-metrics-server-audit-profiles\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479861 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-client-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479866 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/720074df-0a6f-4778-bb73-1cd424158dc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479895 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-registration-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479940 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479971 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479988 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.479999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.479993 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480013 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-sys-fs\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480018 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480030 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fccb02-6982-45a8-9d67-fe60097a84c3-tmp-dir\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480048 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-netns\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480068 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-metrics-server-audit-profiles\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480100 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-agent-certs\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480119 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480139 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2fccb02-6982-45a8-9d67-fe60097a84c3-tmp-dir\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480134 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2fccb02-6982-45a8-9d67-fe60097a84c3-config-volume\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480174 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480185 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-etc-kubernetes\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480197 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480207 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-client-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480202 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480255 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480283 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480315 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3877d90b-6949-48e6-bdd5-568a248a31d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480345 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480345 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/720074df-0a6f-4778-bb73-1cd424158dc9-data-volume\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480389 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-tls\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480392 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480396 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-agent-certs\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480409 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/720074df-0a6f-4778-bb73-1cd424158dc9-data-volume\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480416 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480443 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480456 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2fccb02-6982-45a8-9d67-fe60097a84c3-config-volume\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480461 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480488 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-netns\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480518 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480545 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysconfig\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480554 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21dcf34f-5308-47fd-a7a2-5a277aa95daf-original-pull-secret\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480571 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3877d90b-6949-48e6-bdd5-568a248a31d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480571 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.480585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480606 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-tls\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480616 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-client-certs\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480639 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480646 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-kubelet-config\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480643 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480663 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480685 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-systemd-units\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480749 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3877d90b-6949-48e6-bdd5-568a248a31d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480783 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480810 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-bin\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480833 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-conf\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480850 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-lib-modules\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480864 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/720074df-0a6f-4778-bb73-1cd424158dc9-crio-socket\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480862 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7496f3c4-d42d-446d-8c0e-32272aa84f34-secret-metrics-server-client-certs\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480933 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7pb\" (UniqueName: \"kubernetes.io/projected/996da354-a8e9-4600-9cad-e8beba46a6ab-kube-api-access-wn7pb\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480964 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.480999 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481033 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481051 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481107 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481142 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mj8\" (UniqueName: \"kubernetes.io/projected/2bfc67d0-5890-47b5-9885-4338121de31e-kube-api-access-k4mj8\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481172 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481216 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-federate-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481280 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-dbus\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481311 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-env-overrides\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481325 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481338 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4229762d-c3ea-43fd-851d-506bba6c76db-ovn-node-metrics-cert\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481325 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481376 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481386 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d658122-115c-4a45-965d-5d75e43ff1be-iptables-alerter-script\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481415 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-serving-certs-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481484 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-env-overrides\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481492 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-federate-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481505 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-var-lib-kubelet\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481586 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4229762d-c3ea-43fd-851d-506bba6c76db-ovn-node-metrics-cert\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481602 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481623 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481640 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-kubelet\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481649 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-serving-certs-ca-bundle\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481658 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481649 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d658122-115c-4a45-965d-5d75e43ff1be-iptables-alerter-script\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481676 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481738 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481785 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481820 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481830 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqg7g\" (UniqueName: \"kubernetes.io/projected/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-kube-api-access-sqg7g\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481831 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481927 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481931 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.481988 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482003 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482005 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-system-cni-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.482025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482043 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482072 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482137 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482162 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482194 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482226 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482227 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-multus\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482284 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-serviceca\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482328 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-wtmp\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482345 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-telemeter-client-tls\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482378 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482404 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-etc-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482429 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-script-lib\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482475 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482504 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxqf\" (UniqueName: \"kubernetes.io/projected/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-api-access-9rxqf\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482530 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-kubelet\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482558 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482585 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e7aad03-570e-43d0-bde7-83dd1740ac78-metrics-client-ca\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482625 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482628 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482655 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-systemd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482692 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4229762d-c3ea-43fd-851d-506bba6c76db-ovnkube-script-lib\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482692 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-log-socket\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482739 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-tmp-dir\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482782 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmz2\" (UniqueName: \"kubernetes.io/projected/2d658122-115c-4a45-965d-5d75e43ff1be-kube-api-access-csmz2\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482811 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e7aad03-570e-43d0-bde7-83dd1740ac78-metrics-client-ca\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482820 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-tmp-dir\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482830 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482910 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-system-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482927 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-serviceca\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.482932 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95n8\" (UniqueName: \"kubernetes.io/projected/720074df-0a6f-4778-bb73-1cd424158dc9-kube-api-access-w95n8\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483041 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-sys\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483077 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnkq\" (UniqueName: \"kubernetes.io/projected/363442c0-a599-49dd-a2d2-a0f93ac8c859-kube-api-access-xfnkq\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483106 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483122 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483177 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-grpc-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483211 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483230 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-device-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483294 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483299 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483297 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483387 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7496f3c4-d42d-446d-8c0e-32272aa84f34-audit-log\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483411 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7496f3c4-d42d-446d-8c0e-32272aa84f34-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483429 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-socket-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483460 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483484 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-grpc-tls\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483490 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7496f3c4-d42d-446d-8c0e-32272aa84f34-audit-log\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483514 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn5q\" (UniqueName: \"kubernetes.io/projected/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-kube-api-access-bdn5q\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483524 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483544 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-multus-certs\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483565 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483619 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/363442c0-a599-49dd-a2d2-a0f93ac8c859-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483651 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-netd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483670 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-systemd\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483697 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-metrics-client-ca\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483729 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-cert\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483836 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483855 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/985581c6-d015-4770-992d-b75ab7f0654a-metrics-client-ca\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483909 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483980 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.483990 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfbd\" (UniqueName: \"kubernetes.io/projected/506349da-6a0c-482a-a953-3eaa76e552b6-kube-api-access-5rfbd\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484022 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/720074df-0a6f-4778-bb73-1cd424158dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484026 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-cert\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484050 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484081 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s7f2\" (UniqueName: \"kubernetes.io/projected/0a04c0a7-b90d-4233-bc46-b8430095ef21-kube-api-access-6s7f2\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484090 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484107 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484209 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-root\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484232 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk4dz\" (UniqueName: \"kubernetes.io/projected/7496f3c4-d42d-446d-8c0e-32272aa84f34-kube-api-access-tk4dz\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484249 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/720074df-0a6f-4778-bb73-1cd424158dc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484249 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484267 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484289 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.484256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484292 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2fccb02-6982-45a8-9d67-fe60097a84c3-metrics-tls\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484353 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-var-lib-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484379 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdj2\" (UniqueName: \"kubernetes.io/projected/4229762d-c3ea-43fd-851d-506bba6c76db-kube-api-access-zwdj2\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484409 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484512 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484542 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2fccb02-6982-45a8-9d67-fe60097a84c3-metrics-tls\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484577 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484583 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484613 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-hostroot\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484630 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-host\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484644 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bfc67d0-5890-47b5-9885-4338121de31e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484677 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-ovn\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484706 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-node-log\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484712 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484736 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/996da354-a8e9-4600-9cad-e8beba46a6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484764 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzhk\" (UniqueName: \"kubernetes.io/projected/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-kube-api-access-hhzhk\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484790 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484834 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-cnibin\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484862 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484901 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-konnectivity-ca\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484927 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd62m\" (UniqueName: \"kubernetes.io/projected/7e7aad03-570e-43d0-bde7-83dd1740ac78-kube-api-access-gd62m\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484958 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.484998 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e7aad03-570e-43d0-bde7-83dd1740ac78-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485021 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485043 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485055 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/996da354-a8e9-4600-9cad-e8beba46a6ab-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485059 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d658122-115c-4a45-965d-5d75e43ff1be-host-slash\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485085 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485112 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-tuned\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485116 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485125 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b8f39f3-c5f2-4cde-8a5d-05cd13829521-konnectivity-ca\") pod \"konnectivity-agent-9q452\" (UID: \"0b8f39f3-c5f2-4cde-8a5d-05cd13829521\") " pod="kube-system/konnectivity-agent-9q452" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485146 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485175 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-metrics-client-ca\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485223 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a04c0a7-b90d-4233-bc46-b8430095ef21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485236 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485241 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485296 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-tuned\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485300 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-multus-daemon-config\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485399 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485414 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a04c0a7-b90d-4233-bc46-b8430095ef21-metrics-client-ca\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485423 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-slash\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485532 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/506349da-6a0c-482a-a953-3eaa76e552b6-multus-daemon-config\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485529 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc67d0-5890-47b5-9885-4338121de31e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485570 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485603 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-socket-dir-parent\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485605 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12073275-f042-497b-97cc-10cf8d661a68-metrics-certs\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485629 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-k8s-cni-cncf-io\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485658 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-conf-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485681 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kg9f\" (UniqueName: \"kubernetes.io/projected/12073275-f042-497b-97cc-10cf8d661a68-kube-api-access-8kg9f\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485683 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bfc67d0-5890-47b5-9885-4338121de31e-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485705 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-hosts-file\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485797 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-kubernetes\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485811 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485838 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e7aad03-570e-43d0-bde7-83dd1740ac78-tmp\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.485945 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.486016 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e7aad03-570e-43d0-bde7-83dd1740ac78-tmp\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.486168 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.537475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537393 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nd2\" (UniqueName: \"kubernetes.io/projected/e2fccb02-6982-45a8-9d67-fe60097a84c3-kube-api-access-n9nd2\") pod \"dns-default-qx48k\" (UID: \"e2fccb02-6982-45a8-9d67-fe60097a84c3\") " pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:29.537614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537486 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcl5\" (UniqueName: \"kubernetes.io/projected/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kube-api-access-nfcl5\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.537614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537502 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpz4\" (UniqueName: \"kubernetes.io/projected/b29ee7c6-efd8-4404-8a4d-de677037375e-kube-api-access-twpz4\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.537688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537615 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqg7g\" (UniqueName: \"kubernetes.io/projected/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-kube-api-access-sqg7g\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.537688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537647 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbvb\" (UniqueName: \"kubernetes.io/projected/985581c6-d015-4770-992d-b75ab7f0654a-kube-api-access-4dbvb\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.537688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537650 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7tz\" (UniqueName: \"kubernetes.io/projected/3877d90b-6949-48e6-bdd5-568a248a31d9-kube-api-access-9x7tz\") pod \"openshift-state-metrics-7bd7489957-wkx6z\" (UID: \"3877d90b-6949-48e6-bdd5-568a248a31d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wkx6z" Oct 30 21:17:29.537827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537697 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mj8\" (UniqueName: \"kubernetes.io/projected/2bfc67d0-5890-47b5-9885-4338121de31e-kube-api-access-k4mj8\") pod \"prometheus-operator-9cd89b74c-2mcc4\" (UID: \"2bfc67d0-5890-47b5-9885-4338121de31e\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2mcc4" Oct 30 21:17:29.537827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.537811 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7pb\" (UniqueName: \"kubernetes.io/projected/996da354-a8e9-4600-9cad-e8beba46a6ab-kube-api-access-wn7pb\") pod \"monitoring-plugin-77f4f76594-tbk6f\" (UID: \"996da354-a8e9-4600-9cad-e8beba46a6ab\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:29.539784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.539763 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") pod \"console-75748b998b-p9kst\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:29.539983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.539968 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxqf\" (UniqueName: \"kubernetes.io/projected/2cb13caf-5f94-4a1a-a865-8f3a1ee584fe-kube-api-access-9rxqf\") pod \"kube-state-metrics-5dc8d7d89b-6jmll\" (UID: \"2cb13caf-5f94-4a1a-a865-8f3a1ee584fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-6jmll" Oct 30 21:17:29.540838 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.540818 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95n8\" (UniqueName: \"kubernetes.io/projected/720074df-0a6f-4778-bb73-1cd424158dc9-kube-api-access-w95n8\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.541008 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.540991 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") pod \"console-85894779cf-c48vt\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:29.541067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.540997 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmz2\" (UniqueName: \"kubernetes.io/projected/2d658122-115c-4a45-965d-5d75e43ff1be-kube-api-access-csmz2\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.544800 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.544740 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kg9f\" (UniqueName: \"kubernetes.io/projected/12073275-f042-497b-97cc-10cf8d661a68-kube-api-access-8kg9f\") pod \"network-metrics-daemon-kzs5d\" (UID: \"12073275-f042-497b-97cc-10cf8d661a68\") " pod="openshift-multus/network-metrics-daemon-kzs5d" Oct 30 21:17:29.544924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.544860 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzhk\" (UniqueName: \"kubernetes.io/projected/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-kube-api-access-hhzhk\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.544924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.544909 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn5q\" (UniqueName: \"kubernetes.io/projected/4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd-kube-api-access-bdn5q\") pod \"ingress-canary-ndqrr\" (UID: \"4e6a500a-0ebe-4a44-ba07-8b33ad54c2dd\") " pod="openshift-ingress-canary/ingress-canary-ndqrr" Oct 30 21:17:29.545054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.544956 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4c44\" (UniqueName: \"kubernetes.io/projected/aa327463-70c6-4971-9554-b2df6bce21e5-kube-api-access-j4c44\") pod \"network-check-target-h8c58\" (UID: \"aa327463-70c6-4971-9554-b2df6bce21e5\") " pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:29.545054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.545012 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdj2\" (UniqueName: \"kubernetes.io/projected/4229762d-c3ea-43fd-851d-506bba6c76db-kube-api-access-zwdj2\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.545054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.545017 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") pod \"alertmanager-main-0\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:17:29.545154 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.545055 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk4dz\" (UniqueName: \"kubernetes.io/projected/7496f3c4-d42d-446d-8c0e-32272aa84f34-kube-api-access-tk4dz\") pod \"metrics-server-6dcc758dc7-djcnn\" (UID: \"7496f3c4-d42d-446d-8c0e-32272aa84f34\") " pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:29.545154 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.545104 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd62m\" (UniqueName: \"kubernetes.io/projected/7e7aad03-570e-43d0-bde7-83dd1740ac78-kube-api-access-gd62m\") pod \"thanos-querier-6bfd8c9449-t7f9b\" (UID: \"7e7aad03-570e-43d0-bde7-83dd1740ac78\") " pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:29.546056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.546036 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s7f2\" (UniqueName: \"kubernetes.io/projected/0a04c0a7-b90d-4233-bc46-b8430095ef21-kube-api-access-6s7f2\") pod \"telemeter-client-8dcd9788c-w4vk7\" (UID: \"0a04c0a7-b90d-4233-bc46-b8430095ef21\") " pod="openshift-monitoring/telemeter-client-8dcd9788c-w4vk7" Oct 30 21:17:29.546144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.546129 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfbd\" (UniqueName: \"kubernetes.io/projected/506349da-6a0c-482a-a953-3eaa76e552b6-kube-api-access-5rfbd\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.546212 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.546197 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnkq\" (UniqueName: \"kubernetes.io/projected/363442c0-a599-49dd-a2d2-a0f93ac8c859-kube-api-access-xfnkq\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.587005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.586970 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.587005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587004 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-host\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587025 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-registration-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587041 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-sys-fs\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587103 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-registration-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587108 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-host\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587128 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-netns\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587160 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-sys-fs\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587160 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-netns\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587162 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-etc-kubernetes\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587183 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-etc-kubernetes\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587108 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.587194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587196 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587217 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-netns\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587233 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysconfig\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587248 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587249 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-kubelet-config\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587277 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-kubelet-config\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587283 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-netns\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587282 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-systemd-units\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587333 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysconfig\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587343 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-systemd-units\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587352 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-bin\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587382 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-conf\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587392 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-bin\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587419 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-lib-modules\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587455 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/720074df-0a6f-4778-bb73-1cd424158dc9-crio-socket\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.587490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587489 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/720074df-0a6f-4778-bb73-1cd424158dc9-crio-socket\") pod \"insights-runtime-extractor-7bckf\" (UID: \"720074df-0a6f-4778-bb73-1cd424158dc9\") " pod="openshift-insights/insights-runtime-extractor-7bckf" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587501 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-conf\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587519 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-lib-modules\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587527 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587546 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-dbus\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587564 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-var-lib-kubelet\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587580 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-kubelet\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587598 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-system-cni-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587608 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-run-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587616 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-kubelet\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587623 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587647 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-multus\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587658 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-var-lib-kubelet\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587663 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587660 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21dcf34f-5308-47fd-a7a2-5a277aa95daf-dbus\") pod \"global-pull-secret-syncer-5vztw\" (UID: \"21dcf34f-5308-47fd-a7a2-5a277aa95daf\") " pod="kube-system/global-pull-secret-syncer-5vztw" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587667 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-system-cni-dir\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587673 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-wtmp\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587691 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-multus\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587723 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-node-exporter-wtmp\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587759 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-etc-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587827 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-kubelet\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587845 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587861 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587860 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-kubelet\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587826 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-etc-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587905 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-systemd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587915 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587931 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-systemd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587932 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587931 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-log-socket\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587962 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-log-socket\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587967 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-system-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.587994 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-system-cni-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588001 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-sys\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588008 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-sys\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588028 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-device-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588055 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-socket-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.588061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588081 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-multus-certs\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588105 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-netd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588124 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-device-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588141 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-multus-certs\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588142 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-socket-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588157 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-cni-netd\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588131 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-systemd\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588202 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-root\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588216 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-systemd\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588222 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588227 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/985581c6-d015-4770-992d-b75ab7f0654a-root\") pod \"node-exporter-qz9gs\" (UID: \"985581c6-d015-4770-992d-b75ab7f0654a\") " pod="openshift-monitoring/node-exporter-qz9gs" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588250 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59caf8ab-703a-4d27-bf2c-faa75cdf4c36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmd6c\" (UID: \"59caf8ab-703a-4d27-bf2c-faa75cdf4c36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmd6c" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588251 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-var-lib-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588281 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-hostroot\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588284 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-var-lib-openvswitch\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588296 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-host\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588312 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-ovn\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588323 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-hostroot\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588327 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-node-log\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588345 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-node-log\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588356 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-cnibin\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588373 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2a1b8a3-1ae3-4724-ba2e-7260a79c6812-host\") pod \"node-ca-5bz2b\" (UID: \"e2a1b8a3-1ae3-4724-ba2e-7260a79c6812\") " pod="openshift-image-registry/node-ca-5bz2b" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588384 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-run-ovn\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588385 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d658122-115c-4a45-965d-5d75e43ff1be-host-slash\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588402 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/363442c0-a599-49dd-a2d2-a0f93ac8c859-cnibin\") pod \"multus-additional-cni-plugins-d2f8g\" (UID: \"363442c0-a599-49dd-a2d2-a0f93ac8c859\") " pod="openshift-multus/multus-additional-cni-plugins-d2f8g" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588409 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588437 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-slash\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588458 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d658122-115c-4a45-965d-5d75e43ff1be-host-slash\") pod \"iptables-alerter-5m86j\" (UID: \"2d658122-115c-4a45-965d-5d75e43ff1be\") " pod="openshift-network-operator/iptables-alerter-5m86j" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588461 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-socket-dir-parent\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588495 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4229762d-c3ea-43fd-851d-506bba6c76db-host-slash\") pod \"ovnkube-node-69lqx\" (UID: \"4229762d-c3ea-43fd-851d-506bba6c76db\") " pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588502 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-socket-dir-parent\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588507 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-k8s-cni-cncf-io\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588529 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-sysctl-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588532 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-run-k8s-cni-cncf-io\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588538 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-conf-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588563 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-hosts-file\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588583 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-multus-conf-dir\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588586 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-kubernetes\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588619 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f0fea2c-ba27-4a1c-a48d-c070d9577b1b-hosts-file\") pod \"node-resolver-67mbf\" (UID: \"1f0fea2c-ba27-4a1c-a48d-c070d9577b1b\") " pod="openshift-dns/node-resolver-67mbf" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588616 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-cnibin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588649 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-kubernetes\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588649 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-bin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588663 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-cnibin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588672 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506349da-6a0c-482a-a953-3eaa76e552b6-host-var-lib-cni-bin\") pod \"multus-t6fc6\" (UID: \"506349da-6a0c-482a-a953-3eaa76e552b6\") " pod="openshift-multus/multus-t6fc6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588675 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-modprobe-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588701 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-run\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588733 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-etc-modprobe-d\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588752 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-run\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588755 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-sys\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.589347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.588812 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b29ee7c6-efd8-4404-8a4d-de677037375e-sys\") pod \"tuned-x6zm6\" (UID: \"b29ee7c6-efd8-4404-8a4d-de677037375e\") " pod="openshift-cluster-node-tuning-operator/tuned-x6zm6" Oct 30 21:17:29.966366 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.966330 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:29.970180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:29.970152 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tbk6f" Oct 30 21:17:30.729690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:30.728598 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:30.951715 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:30.951676 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75748b998b-p9kst"] Oct 30 21:17:31.205420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:31.205395 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qx48k" Oct 30 21:17:32.070591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.070557 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:32.382185 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.382154 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:32.437681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.437652 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:32.445427 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.445341 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bfd8c9449-t7f9b" Oct 30 21:17:32.577646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.577612 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:32.581318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.581294 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:32.773295 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:32.773222 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:17:34.566690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:34.566656 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:36.979678 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:36.979598 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:36.980077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:36.979869 5914 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:17:38.508315 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:38.508282 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:42.344601 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:42.344566 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:17:42.387594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:42.387561 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:42.391156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:42.391135 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:17:47.526011 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:47.525974 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:48.483114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:48.483083 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-69lqx" Oct 30 21:17:52.075899 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:52.075854 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:52.080130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:52.080106 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dcc758dc7-djcnn" Oct 30 21:17:55.972760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:55.972689 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-75748b998b-p9kst" podUID="39ab801c-ab61-43bb-8523-8798c89b5279" containerName="console" containerID="cri-o://52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340" gracePeriod=15 Oct 30 21:17:56.273242 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.273215 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:56.402042 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.402012 5914 generic.go:356] "Generic (PLEG): container finished" podID="39ab801c-ab61-43bb-8523-8798c89b5279" containerID="52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340" exitCode=0 Oct 30 21:17:56.402203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.402057 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75748b998b-p9kst" Oct 30 21:17:56.402203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.402108 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75748b998b-p9kst" event={"ID":"39ab801c-ab61-43bb-8523-8798c89b5279","Type":"ContainerDied","Data":"52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340"} Oct 30 21:17:56.402203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.402172 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75748b998b-p9kst" event={"ID":"39ab801c-ab61-43bb-8523-8798c89b5279","Type":"ContainerDied","Data":"64fff64eb4eb407c15b0416fce3d087305e7d1326dba9ca5b1255b5571b57dee"} Oct 30 21:17:56.402328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.402204 5914 scope.go:122] "RemoveContainer" containerID="52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340" Oct 30 21:17:56.411928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.411907 5914 scope.go:122] "RemoveContainer" containerID="52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340" Oct 30 21:17:56.412202 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:17:56.412176 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340\": container with ID starting with 52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340 not found: ID does not exist" containerID="52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340" Oct 30 21:17:56.412325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.412207 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340"} err="failed to get container status \"52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340\": rpc error: code = NotFound desc = could not find container \"52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340\": container with ID starting with 52a72db9fbac201fba45d499b232bed01b383a5866e64f8ff10763264192a340 not found: ID does not exist" Oct 30 21:17:56.441521 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441495 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.441647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441560 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.441647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441582 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.441647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441603 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.441811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441647 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.441811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441688 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.441811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441734 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:56.441811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.441801 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") pod \"39ab801c-ab61-43bb-8523-8798c89b5279\" (UID: \"39ab801c-ab61-43bb-8523-8798c89b5279\") " Oct 30 21:17:56.442190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.442137 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:56.442190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.442167 5914 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39ab801c-ab61-43bb-8523-8798c89b5279-tmp\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.442190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.442168 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:56.442352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.442195 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:56.444157 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.444134 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:56.444261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.444189 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "kube-api-access-xtzg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:56.444261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.444207 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert" pod "39ab801c-ab61-43bb-8523-8798c89b5279" (UID: "39ab801c-ab61-43bb-8523-8798c89b5279"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:56.543593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.543522 5914 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-oauth-serving-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.543593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.543548 5914 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.543593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.543559 5914 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-serving-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.543593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.543571 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzg8\" (UniqueName: \"kubernetes.io/projected/39ab801c-ab61-43bb-8523-8798c89b5279-kube-api-access-xtzg8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.543593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.543582 5914 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/39ab801c-ab61-43bb-8523-8798c89b5279-console-oauth-config\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.543593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.543591 5914 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/39ab801c-ab61-43bb-8523-8798c89b5279-console-config\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:17:56.821071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.820980 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75748b998b-p9kst"] Oct 30 21:17:56.870650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:56.870613 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75748b998b-p9kst"] Oct 30 21:17:57.169194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:57.169166 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9q452" Oct 30 21:17:58.214968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:17:58.214934 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ab801c-ab61-43bb-8523-8798c89b5279" path="/var/lib/kubelet/pods/39ab801c-ab61-43bb-8523-8798c89b5279/volumes" Oct 30 21:18:02.776330 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:02.776294 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h8c58" Oct 30 21:18:12.344564 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:12.344521 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:18:30.945737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.945703 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:18:30.946216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.945984 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="alertmanager" containerID="cri-o://ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" gracePeriod=120 Oct 30 21:18:30.946216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.946068 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-web" containerID="cri-o://518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" gracePeriod=120 Oct 30 21:18:30.946216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.946065 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-metric" containerID="cri-o://44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" gracePeriod=120 Oct 30 21:18:30.946216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.946121 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="config-reloader" containerID="cri-o://94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" gracePeriod=120 Oct 30 21:18:30.946216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.946128 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="prom-label-proxy" containerID="cri-o://28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" gracePeriod=120 Oct 30 21:18:30.946498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:30.946283 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy" containerID="cri-o://f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" gracePeriod=120 Oct 30 21:18:31.199541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.199485 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.326738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.326705 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.326738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.326744 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.327045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.326786 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.327045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.326851 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.327045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.326942 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.327045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.326971 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.327045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327007 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.327045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327044 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327080 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327089 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327122 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327181 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327215 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.327244 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") pod \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\" (UID: \"e9e92bbc-f24d-4a92-b28f-496dbeef61ce\") " Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.328048 5914 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-main-db\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.328994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.328496 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:31.330024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.329741 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:31.331055 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.331022 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.331152 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.331053 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "kube-api-access-cdrvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:31.332098 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.332070 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.332814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.332787 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.333039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.333013 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.333252 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.333232 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:31.333475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.333456 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.334393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.334371 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:18:31.335709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.335681 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.343068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.343041 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config" pod "e9e92bbc-f24d-4a92-b28f-496dbeef61ce" (UID: "e9e92bbc-f24d-4a92-b28f-496dbeef61ce"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:31.428668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428637 5914 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-cluster-tls-config\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428666 5914 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428675 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-metrics-client-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428685 5914 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-main-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428696 5914 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-config-out\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428706 5914 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428715 5914 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428724 5914 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428734 5914 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-tls-assets\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428743 5914 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428752 5914 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-web-config\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.428908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.428761 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrvp\" (UniqueName: \"kubernetes.io/projected/e9e92bbc-f24d-4a92-b28f-496dbeef61ce-kube-api-access-cdrvp\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534817 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" exitCode=0 Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534839 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" exitCode=0 Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534848 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" exitCode=0 Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534854 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" exitCode=0 Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534861 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" exitCode=0 Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534867 5914 generic.go:356] "Generic (PLEG): container finished" podID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" exitCode=0 Oct 30 21:18:31.534917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534909 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534923 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.534979 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.535008 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.535033 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.535045 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.535058 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.535084 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} Oct 30 21:18:31.535253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.535107 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e9e92bbc-f24d-4a92-b28f-496dbeef61ce","Type":"ContainerDied","Data":"ec977eb79a06132fc6b5b1b48b8994940e4e81b11d8d71f63e890b0a87f6236c"} Oct 30 21:18:31.543528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.543510 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.550720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.550703 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.557276 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.557261 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.563825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.563807 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.572130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.572115 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.579321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.579305 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.586193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586177 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.586469 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.586448 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.586522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586481 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} err="failed to get container status \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" Oct 30 21:18:31.586522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586502 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.586713 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.586693 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.586802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586717 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} err="failed to get container status \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" Oct 30 21:18:31.586802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586734 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.586959 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.586941 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.587000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586963 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} err="failed to get container status \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" Oct 30 21:18:31.587000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.586974 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.587217 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.587199 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.587280 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.587225 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} err="failed to get container status \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" Oct 30 21:18:31.587280 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.587244 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.587457 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.587440 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.587500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.587463 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} err="failed to get container status \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" Oct 30 21:18:31.587500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.587479 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.587717 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.587697 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.587773 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.587723 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} err="failed to get container status \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" Oct 30 21:18:31.587773 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.587734 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.588052 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:31.588032 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.588131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588060 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} err="failed to get container status \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" Oct 30 21:18:31.588131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588079 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.588358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588337 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} err="failed to get container status \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" Oct 30 21:18:31.588421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588359 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.588570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588555 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} err="failed to get container status \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" Oct 30 21:18:31.588626 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588571 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.588773 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588754 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} err="failed to get container status \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" Oct 30 21:18:31.588834 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588774 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.589006 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.588986 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} err="failed to get container status \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" Oct 30 21:18:31.589090 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589007 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.589218 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589202 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} err="failed to get container status \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" Oct 30 21:18:31.589259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589219 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.589443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589424 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} err="failed to get container status \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" Oct 30 21:18:31.589480 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589444 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.589661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589644 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} err="failed to get container status \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" Oct 30 21:18:31.589705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589662 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.589874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589858 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} err="failed to get container status \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" Oct 30 21:18:31.589944 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.589875 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.590111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590094 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} err="failed to get container status \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" Oct 30 21:18:31.590154 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590113 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.590318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590301 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} err="failed to get container status \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" Oct 30 21:18:31.590360 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590318 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.590500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590482 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} err="failed to get container status \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" Oct 30 21:18:31.590547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590500 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.590683 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590666 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} err="failed to get container status \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" Oct 30 21:18:31.590722 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590684 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.590899 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590866 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} err="failed to get container status \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" Oct 30 21:18:31.590899 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.590894 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.591096 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591073 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} err="failed to get container status \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" Oct 30 21:18:31.591139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591096 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.591288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591274 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} err="failed to get container status \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" Oct 30 21:18:31.591326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591288 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.591469 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591455 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} err="failed to get container status \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" Oct 30 21:18:31.591544 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591469 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.591653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591638 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} err="failed to get container status \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" Oct 30 21:18:31.591695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591652 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.591831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591816 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} err="failed to get container status \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" Oct 30 21:18:31.591947 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.591831 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.592075 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592060 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} err="failed to get container status \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" Oct 30 21:18:31.592112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592075 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.592257 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592242 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} err="failed to get container status \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" Oct 30 21:18:31.592304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592257 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.592443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592429 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} err="failed to get container status \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" Oct 30 21:18:31.592480 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592443 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.592622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592608 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} err="failed to get container status \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" Oct 30 21:18:31.592622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592621 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.592809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592795 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} err="failed to get container status \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" Oct 30 21:18:31.592809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592809 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.593000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592984 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} err="failed to get container status \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" Oct 30 21:18:31.593000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.592999 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.593183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593169 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} err="failed to get container status \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" Oct 30 21:18:31.593183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593182 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.593358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593343 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} err="failed to get container status \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" Oct 30 21:18:31.593402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593358 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.593536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593513 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} err="failed to get container status \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" Oct 30 21:18:31.593581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593536 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.593717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593702 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} err="failed to get container status \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" Oct 30 21:18:31.593754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593716 5914 scope.go:122] "RemoveContainer" containerID="28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1" Oct 30 21:18:31.593895 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593863 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1"} err="failed to get container status \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": rpc error: code = NotFound desc = could not find container \"28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1\": container with ID starting with 28025b6dde9dd27f2e16a6d91b2f90227fa308279be0bd59bb212eab63e146e1 not found: ID does not exist" Oct 30 21:18:31.593934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.593897 5914 scope.go:122] "RemoveContainer" containerID="44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b" Oct 30 21:18:31.594094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594079 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b"} err="failed to get container status \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": rpc error: code = NotFound desc = could not find container \"44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b\": container with ID starting with 44dd7979508a081e49ecd0465edfac2327acedd0a19913cf28612095e277b10b not found: ID does not exist" Oct 30 21:18:31.594094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594092 5914 scope.go:122] "RemoveContainer" containerID="f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499" Oct 30 21:18:31.594302 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594283 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499"} err="failed to get container status \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": rpc error: code = NotFound desc = could not find container \"f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499\": container with ID starting with f7bd191978d44b7fde64b8a115e6476363dbe17e97d353a271e0df44e96f0499 not found: ID does not exist" Oct 30 21:18:31.594363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594303 5914 scope.go:122] "RemoveContainer" containerID="518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f" Oct 30 21:18:31.594549 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594529 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f"} err="failed to get container status \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": rpc error: code = NotFound desc = could not find container \"518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f\": container with ID starting with 518daa2da2ab3a96db83c2c46c45462ffde1c1cf8e27210c6ba325b97916ff7f not found: ID does not exist" Oct 30 21:18:31.594603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594550 5914 scope.go:122] "RemoveContainer" containerID="94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97" Oct 30 21:18:31.594778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594754 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97"} err="failed to get container status \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": rpc error: code = NotFound desc = could not find container \"94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97\": container with ID starting with 94e6757463d93344a38c6d559058e780676189f08e4465b9c02c7cc31c31fc97 not found: ID does not exist" Oct 30 21:18:31.594848 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.594781 5914 scope.go:122] "RemoveContainer" containerID="ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be" Oct 30 21:18:31.595043 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.595028 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be"} err="failed to get container status \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": rpc error: code = NotFound desc = could not find container \"ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be\": container with ID starting with ae13f84192af1f1b287d915b2e89bb15fb4404ba301e369e59ceffab8da371be not found: ID does not exist" Oct 30 21:18:31.595091 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.595054 5914 scope.go:122] "RemoveContainer" containerID="d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb" Oct 30 21:18:31.595260 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.595241 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb"} err="failed to get container status \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": rpc error: code = NotFound desc = could not find container \"d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb\": container with ID starting with d8009b5979de00fa1c086a5de7e52222936924f1ba60a1562f650853e789e3fb not found: ID does not exist" Oct 30 21:18:31.701907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.701491 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755008 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755069 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755798 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="prom-label-proxy" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755820 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="prom-label-proxy" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755835 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755848 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755864 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="config-reloader" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755871 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="config-reloader" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755933 5914 cpu_manager.go:414] "containerMap: removing container" podUID="39ab801c-ab61-43bb-8523-8798c89b5279" containerName="console" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755941 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ab801c-ab61-43bb-8523-8798c89b5279" containerName="console" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755989 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-metric" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.755997 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-metric" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756012 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="init-config-reloader" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756019 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="init-config-reloader" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756075 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-web" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756090 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-web" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756100 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="alertmanager" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756106 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="alertmanager" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756252 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-metric" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756267 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="init-config-reloader" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756281 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy-web" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756288 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="alertmanager" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756298 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="prom-label-proxy" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756305 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="kube-rbac-proxy" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756320 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" containerName="config-reloader" Oct 30 21:18:31.757281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.756330 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ab801c-ab61-43bb-8523-8798c89b5279" containerName="console" Oct 30 21:18:31.762876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.762855 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.820709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820627 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 21:18:31.820709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820662 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 21:18:31.820709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820662 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 21:18:31.821027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820710 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 21:18:31.821027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820745 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 21:18:31.821027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820779 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 21:18:31.821027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820866 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-57txw\"" Oct 30 21:18:31.821027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.820921 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 21:18:31.821205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.821088 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 21:18:31.832601 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832575 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832614 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832636 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832734 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832709 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832752 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-config-out\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832771 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832830 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832899 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832930 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832982 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832945 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjl2\" (UniqueName: \"kubernetes.io/projected/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-kube-api-access-wpjl2\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.832982 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832962 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-web-config\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.833043 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.832984 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.833075 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.833061 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.866253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.866221 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 21:18:31.871295 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.871274 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:18:31.934453 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934420 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934462 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934486 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934511 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934540 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-config-out\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934556 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934580 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934615 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934649 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.934676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934668 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjl2\" (UniqueName: \"kubernetes.io/projected/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-kube-api-access-wpjl2\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.935144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934698 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-web-config\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.935144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934735 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.935144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934775 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.935144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.934873 5914 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:18:31.935363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.935165 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.936334 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.936265 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.936696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.936625 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.937875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.937852 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.938017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.937910 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.938348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.938329 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-config-out\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.938493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.938472 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-web-config\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.938542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.938527 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.938650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.938616 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.939442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.939408 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.939674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.939656 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.939989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.939970 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:31.994944 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:31.994919 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjl2\" (UniqueName: \"kubernetes.io/projected/8b692b54-76d1-4a1b-bc43-021c8e55f8c7-kube-api-access-wpjl2\") pod \"alertmanager-main-0\" (UID: \"8b692b54-76d1-4a1b-bc43-021c8e55f8c7\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:32.079769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:32.079689 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 21:18:32.214315 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:32.214278 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e92bbc-f24d-4a92-b28f-496dbeef61ce" path="/var/lib/kubelet/pods/e9e92bbc-f24d-4a92-b28f-496dbeef61ce/volumes" Oct 30 21:18:32.554870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:32.554836 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 21:18:32.558001 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:18:32.557971 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b692b54_76d1_4a1b_bc43_021c8e55f8c7.slice/crio-bdd1d62e04747d56e567d1a667833aa2f62a4891b5a50369541cd62c63142120 WatchSource:0}: Error finding container bdd1d62e04747d56e567d1a667833aa2f62a4891b5a50369541cd62c63142120: Status 404 returned error can't find the container with id bdd1d62e04747d56e567d1a667833aa2f62a4891b5a50369541cd62c63142120 Oct 30 21:18:33.168041 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.168003 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b8f77ff6-f2wdn"] Oct 30 21:18:33.169963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.169942 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.229955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.229924 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8f77ff6-f2wdn"] Oct 30 21:18:33.246482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246449 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-trusted-ca-bundle\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246481 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceb4a026-3d90-456a-8815-123b22c036b8-console-oauth-config\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246634 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246509 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7xg\" (UniqueName: \"kubernetes.io/projected/ceb4a026-3d90-456a-8815-123b22c036b8-kube-api-access-hw7xg\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246682 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246667 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-oauth-serving-cert\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246698 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-console-config\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246751 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246724 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceb4a026-3d90-456a-8815-123b22c036b8-tmp\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246827 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-service-ca\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.246930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.246916 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceb4a026-3d90-456a-8815-123b22c036b8-console-serving-cert\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348017 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceb4a026-3d90-456a-8815-123b22c036b8-console-serving-cert\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348060 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-trusted-ca-bundle\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348076 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceb4a026-3d90-456a-8815-123b22c036b8-console-oauth-config\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348098 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7xg\" (UniqueName: \"kubernetes.io/projected/ceb4a026-3d90-456a-8815-123b22c036b8-kube-api-access-hw7xg\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348138 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-oauth-serving-cert\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348157 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-console-config\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348191 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceb4a026-3d90-456a-8815-123b22c036b8-tmp\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348228 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-service-ca\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.348792 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.348740 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceb4a026-3d90-456a-8815-123b22c036b8-tmp\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.349074 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.349051 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-service-ca\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.349176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.349051 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-console-config\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.349176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.349057 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-oauth-serving-cert\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.349498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.349468 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceb4a026-3d90-456a-8815-123b22c036b8-trusted-ca-bundle\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.350753 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.350731 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ceb4a026-3d90-456a-8815-123b22c036b8-console-serving-cert\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.350836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.350752 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ceb4a026-3d90-456a-8815-123b22c036b8-console-oauth-config\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.405337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.405300 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7xg\" (UniqueName: \"kubernetes.io/projected/ceb4a026-3d90-456a-8815-123b22c036b8-kube-api-access-hw7xg\") pod \"console-5b8f77ff6-f2wdn\" (UID: \"ceb4a026-3d90-456a-8815-123b22c036b8\") " pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.479645 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.479546 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:33.545654 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.545613 5914 generic.go:356] "Generic (PLEG): container finished" podID="8b692b54-76d1-4a1b-bc43-021c8e55f8c7" containerID="599f4a7c354c7842f84db5b5f3ad413a8e871144c66595714e1b9a72d7c24391" exitCode=0 Oct 30 21:18:33.545800 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.545687 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerDied","Data":"599f4a7c354c7842f84db5b5f3ad413a8e871144c66595714e1b9a72d7c24391"} Oct 30 21:18:33.545800 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.545770 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"bdd1d62e04747d56e567d1a667833aa2f62a4891b5a50369541cd62c63142120"} Oct 30 21:18:33.862550 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:33.862525 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8f77ff6-f2wdn"] Oct 30 21:18:33.865397 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:18:33.865371 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb4a026_3d90_456a_8815_123b22c036b8.slice/crio-a725b1b9ad2e0359293f4f84ea0eb995ecdd1829060858d5d2752132d1e0e968 WatchSource:0}: Error finding container a725b1b9ad2e0359293f4f84ea0eb995ecdd1829060858d5d2752132d1e0e968: Status 404 returned error can't find the container with id a725b1b9ad2e0359293f4f84ea0eb995ecdd1829060858d5d2752132d1e0e968 Oct 30 21:18:34.553406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.553366 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"2d857dfe8fb8b0202a6d0261a45bfc64b6b2187d2f5529e088ade396a2830815"} Oct 30 21:18:34.553865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.553456 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"9e2b582aa733268451e211c66b9ca8e98e64332a35595ee620bc891746e05cc3"} Oct 30 21:18:34.553865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.553529 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"ed548e7750e2fa609e9cf1a477d9246fafbce5889fa5cbcc14e79fe57a7b0d2d"} Oct 30 21:18:34.553865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.553567 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"87d642b3d75c5a31d52e94bfd7252dcbdee608d2adffba8576d3f737216be398"} Oct 30 21:18:34.553865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.553608 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"f2757a1ba6c6b2410d66923bc81624e0407ed739c43a67365c6812fbf1ca9d89"} Oct 30 21:18:34.553865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.553648 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b692b54-76d1-4a1b-bc43-021c8e55f8c7","Type":"ContainerStarted","Data":"2c5d54ec61cb0e3f4b001360ffa488a27370d48e581ebf1e67346790773ded47"} Oct 30 21:18:34.555112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.555083 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8f77ff6-f2wdn" event={"ID":"ceb4a026-3d90-456a-8815-123b22c036b8","Type":"ContainerStarted","Data":"5249af7357ec9cde4f4afa50e4a643b91ce7379613ba4bc084f1f51a5d20787e"} Oct 30 21:18:34.555230 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.555131 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8f77ff6-f2wdn" event={"ID":"ceb4a026-3d90-456a-8815-123b22c036b8","Type":"ContainerStarted","Data":"a725b1b9ad2e0359293f4f84ea0eb995ecdd1829060858d5d2752132d1e0e968"} Oct 30 21:18:34.677693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.677639 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.665767389 podStartE2EDuration="3.677625696s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.858307ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:34.676621031 +0000 UTC m=+66.578485097" watchObservedRunningTime="2026-10-30 21:18:34.677625696 +0000 UTC m=+66.579489760" Oct 30 21:18:34.793839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:34.793775 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b8f77ff6-f2wdn" podStartSLOduration=1.7937558660000001 podStartE2EDuration="1.793755866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:34.792720976 +0000 UTC m=+66.694585074" watchObservedRunningTime="2026-10-30 21:18:34.793755866 +0000 UTC m=+66.695619928" Oct 30 21:18:42.344645 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:18:42.344611 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:18:43.480302 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:43.480267 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:43.480686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:43.480321 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:43.484934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:43.484913 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:43.594591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:43.594563 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b8f77ff6-f2wdn" Oct 30 21:18:43.746255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:18:43.746173 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85894779cf-c48vt"] Oct 30 21:19:08.766471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:08.766388 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85894779cf-c48vt" podUID="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" containerName="console" containerID="cri-o://c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447" gracePeriod=15 Oct 30 21:19:09.013916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.013870 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:19:09.056379 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056309 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056379 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056346 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056396 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056444 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056499 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056533 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056601 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.056742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056649 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") pod \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\" (UID: \"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f\") " Oct 30 21:19:09.057676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.057233 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:09.057676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.057669 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:09.057845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.056907 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:09.057845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.057698 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:09.059914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.058103 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:09.063731 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.063682 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:09.064032 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.063912 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "kube-api-access-zmfsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:09.064979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.064902 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert" pod "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" (UID: "0d8caf36-f02d-46e0-a7f9-fec2987fbc0f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:19:09.158778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158749 5914 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-trusted-ca-bundle\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.158778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158775 5914 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-tmp\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.159002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158786 5914 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-serving-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.159002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158814 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfsk\" (UniqueName: \"kubernetes.io/projected/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-kube-api-access-zmfsk\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.159002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158824 5914 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-oauth-serving-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.159002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158833 5914 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-config\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.159002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158841 5914 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.159002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.158849 5914 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f-console-oauth-config\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:19:09.687711 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.687678 5914 generic.go:356] "Generic (PLEG): container finished" podID="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" containerID="c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447" exitCode=0 Oct 30 21:19:09.687949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.687740 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85894779cf-c48vt" Oct 30 21:19:09.687949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.687747 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85894779cf-c48vt" event={"ID":"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f","Type":"ContainerDied","Data":"c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447"} Oct 30 21:19:09.687949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.687797 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85894779cf-c48vt" event={"ID":"0d8caf36-f02d-46e0-a7f9-fec2987fbc0f","Type":"ContainerDied","Data":"5a169ce2bbb9ddb788ac53dd37070492edeb785dfe27a3e8ea9d16aeb3f12500"} Oct 30 21:19:09.687949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.687822 5914 scope.go:122] "RemoveContainer" containerID="c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447" Oct 30 21:19:09.696313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.696296 5914 scope.go:122] "RemoveContainer" containerID="c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447" Oct 30 21:19:09.696583 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:19:09.696566 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447\": container with ID starting with c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447 not found: ID does not exist" containerID="c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447" Oct 30 21:19:09.696632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.696594 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447"} err="failed to get container status \"c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447\": rpc error: code = NotFound desc = could not find container \"c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447\": container with ID starting with c8e11446fa2303e0e5c53702b4b871609d32a20c8dff49008dc0a384b7ae9447 not found: ID does not exist" Oct 30 21:19:09.849945 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.849915 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85894779cf-c48vt"] Oct 30 21:19:09.899588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:09.899557 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85894779cf-c48vt"] Oct 30 21:19:10.213974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:10.213944 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" path="/var/lib/kubelet/pods/0d8caf36-f02d-46e0-a7f9-fec2987fbc0f/volumes" Oct 30 21:19:10.247398 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:19:10.247360 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" podUID="d6458e4c6fa120c378f995c71dc796f0" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:19:12.344411 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:19:12.344372 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:19:42.344118 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:19:42.344079 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:20:12.343747 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:20:12.343709 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:20:42.344807 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:20:42.344722 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:20:50.833977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.833945 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp"] Oct 30 21:20:50.834454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.834368 5914 cpu_manager.go:414] "containerMap: removing container" podUID="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" containerName="console" Oct 30 21:20:50.834454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.834387 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" containerName="console" Oct 30 21:20:50.834527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.834479 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8caf36-f02d-46e0-a7f9-fec2987fbc0f" containerName="console" Oct 30 21:20:50.837500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.837481 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877334 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877300 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877334 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877335 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877374 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877494 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877582 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877528 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877582 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877553 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvv2v\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.877688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.877629 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.896710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.896674 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:20:50.897174 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.897158 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:20:50.897232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.897191 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:20:50.940981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.940951 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:20:50.940981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.940950 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:20:50.941194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.941001 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:20:50.941194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.941002 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bckcj\"" Oct 30 21:20:50.941776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.941753 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp"] Oct 30 21:20:50.978406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978377 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.978520 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978412 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.978520 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978460 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.978520 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978498 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.978616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978528 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.978616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978560 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvv2v\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.978616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.978598 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.979444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.979419 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.980997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.980965 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.981134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.981116 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.981194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.981143 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:50.981194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:50.981125 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:51.036028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:51.036001 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvv2v\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:51.036287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:51.036263 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jlgnp\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:51.148745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:51.148717 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:51.569268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:51.569182 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp"] Oct 30 21:20:51.609914 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:20:51.609871 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00ef032_c2c0_4f88_9c57_775f194c49d1.slice/crio-8ec8eb513e9e866f1df440bf2ecb59ad57d29405b4e98effca60bebf0e08300b WatchSource:0}: Error finding container 8ec8eb513e9e866f1df440bf2ecb59ad57d29405b4e98effca60bebf0e08300b: Status 404 returned error can't find the container with id 8ec8eb513e9e866f1df440bf2ecb59ad57d29405b4e98effca60bebf0e08300b Oct 30 21:20:51.611847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:51.611830 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:20:52.067842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:52.067810 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" event={"ID":"a00ef032-c2c0-4f88-9c57-775f194c49d1","Type":"ContainerStarted","Data":"8ec8eb513e9e866f1df440bf2ecb59ad57d29405b4e98effca60bebf0e08300b"} Oct 30 21:20:55.291389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:55.291354 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:20:55.291813 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:55.291429 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:20:56.085317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:56.085285 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" event={"ID":"a00ef032-c2c0-4f88-9c57-775f194c49d1","Type":"ContainerStarted","Data":"46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8"} Oct 30 21:20:56.085540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:56.085463 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:20:56.087149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:56.087113 5914 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-jlgnp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:20:56.087259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:56.087172 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:20:56.291468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:56.291408 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" podStartSLOduration=2.612916257 podStartE2EDuration="6.291394941s" totalImagesPullingTime="3.678478684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:56.290868532 +0000 UTC m=+208.192732596" watchObservedRunningTime="2026-10-30 21:20:56.291394941 +0000 UTC m=+208.193259005" Oct 30 21:20:57.089529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:57.089491 5914 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-jlgnp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:20:57.089696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:20:57.089536 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:21:00.090446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:00.090413 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:21:12.345052 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:21:12.345013 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:21:34.368343 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.368305 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng"] Oct 30 21:21:34.373452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.373437 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.427594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.427558 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:21:34.427810 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.427785 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:21:34.427932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.427791 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:21:34.428003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.427927 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-929kb\"" Oct 30 21:21:34.477753 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.477716 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:21:34.477956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.477778 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:21:34.479678 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.479653 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng"] Oct 30 21:21:34.554042 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.554008 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.554042 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.554041 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-metrics-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.554250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.554154 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnvq\" (UniqueName: \"kubernetes.io/projected/48249dfb-e1d3-4bfb-90c4-26147cb8359c-kube-api-access-mlnvq\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.554250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.554209 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48249dfb-e1d3-4bfb-90c4-26147cb8359c-manifests\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.554328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.554268 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48249dfb-e1d3-4bfb-90c4-26147cb8359c-tmp\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.554328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.554305 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655240 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48249dfb-e1d3-4bfb-90c4-26147cb8359c-manifests\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655279 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48249dfb-e1d3-4bfb-90c4-26147cb8359c-tmp\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655306 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655339 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655357 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-metrics-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655401 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnvq\" (UniqueName: \"kubernetes.io/projected/48249dfb-e1d3-4bfb-90c4-26147cb8359c-kube-api-access-mlnvq\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.655804 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.655781 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48249dfb-e1d3-4bfb-90c4-26147cb8359c-manifests\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.657975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.657950 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48249dfb-e1d3-4bfb-90c4-26147cb8359c-tmp\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.658223 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.658204 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.658266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.658208 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-metrics-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.658266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.658236 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48249dfb-e1d3-4bfb-90c4-26147cb8359c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.715996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.715961 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnvq\" (UniqueName: \"kubernetes.io/projected/48249dfb-e1d3-4bfb-90c4-26147cb8359c-kube-api-access-mlnvq\") pod \"opendatahub-operator-controller-manager-85bb549b7d-fxlng\" (UID: \"48249dfb-e1d3-4bfb-90c4-26147cb8359c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:34.986817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:34.986732 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:35.521231 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:35.521124 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng"] Oct 30 21:21:35.524342 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:21:35.524312 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48249dfb_e1d3_4bfb_90c4_26147cb8359c.slice/crio-4c94fa235e30c6717df3d38c63af37fec828bf3d6aeac6113591e90878d95798 WatchSource:0}: Error finding container 4c94fa235e30c6717df3d38c63af37fec828bf3d6aeac6113591e90878d95798: Status 404 returned error can't find the container with id 4c94fa235e30c6717df3d38c63af37fec828bf3d6aeac6113591e90878d95798 Oct 30 21:21:36.236059 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:36.236025 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" event={"ID":"48249dfb-e1d3-4bfb-90c4-26147cb8359c","Type":"ContainerStarted","Data":"4c94fa235e30c6717df3d38c63af37fec828bf3d6aeac6113591e90878d95798"} Oct 30 21:21:39.251169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:39.251132 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" event={"ID":"48249dfb-e1d3-4bfb-90c4-26147cb8359c","Type":"ContainerStarted","Data":"486bf751f4690cd2bff042449d6e8a391098da3128c530b6c44f7e43890b894f"} Oct 30 21:21:40.257073 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:40.257042 5914 generic.go:356] "Generic (PLEG): container finished" podID="48249dfb-e1d3-4bfb-90c4-26147cb8359c" containerID="486bf751f4690cd2bff042449d6e8a391098da3128c530b6c44f7e43890b894f" exitCode=0 Oct 30 21:21:40.257499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:40.257081 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" event={"ID":"48249dfb-e1d3-4bfb-90c4-26147cb8359c","Type":"ContainerDied","Data":"486bf751f4690cd2bff042449d6e8a391098da3128c530b6c44f7e43890b894f"} Oct 30 21:21:41.263677 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:41.263646 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" event={"ID":"48249dfb-e1d3-4bfb-90c4-26147cb8359c","Type":"ContainerStarted","Data":"863f466f25dbd891a40b4c147105a4a9d6192340cb0a26392c126fd5a952bc3e"} Oct 30 21:21:41.264127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:41.263771 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:21:41.498747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:41.498696 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" podStartSLOduration=2.717373025 podStartE2EDuration="7.498682475s" totalImagesPullingTime="4.306119073s" totalInitContainerRuntime="475.190377ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:41.498416862 +0000 UTC m=+253.400280933" watchObservedRunningTime="2026-10-30 21:21:41.498682475 +0000 UTC m=+253.400546537" Oct 30 21:21:42.344241 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:21:42.344201 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:21:52.269661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:21:52.269635 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85bb549b7d-fxlng" Oct 30 21:22:12.344492 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:22:12.344401 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:22:28.260784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:28.260757 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:22:28.261749 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:28.261731 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:22:42.347778 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:22:42.347738 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:22:42.676536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.676506 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-578b8bf98-4lglr"] Oct 30 21:22:42.678612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.678588 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.730564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.730531 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:22:42.730734 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.730575 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:22:42.768521 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.768490 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b32414f-a7bf-4d68-bdc7-6568dc26d900-tls-certs\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.768697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.768532 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b32414f-a7bf-4d68-bdc7-6568dc26d900-tmp\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.768697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.768558 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfwm\" (UniqueName: \"kubernetes.io/projected/9b32414f-a7bf-4d68-bdc7-6568dc26d900-kube-api-access-5lfwm\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.780407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.780378 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2hw5t\"" Oct 30 21:22:42.781310 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.781286 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-578b8bf98-4lglr"] Oct 30 21:22:42.869224 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.869191 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfwm\" (UniqueName: \"kubernetes.io/projected/9b32414f-a7bf-4d68-bdc7-6568dc26d900-kube-api-access-5lfwm\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.869420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.869292 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b32414f-a7bf-4d68-bdc7-6568dc26d900-tls-certs\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.869420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.869339 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b32414f-a7bf-4d68-bdc7-6568dc26d900-tmp\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.871712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.871683 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b32414f-a7bf-4d68-bdc7-6568dc26d900-tmp\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.872014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.871994 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b32414f-a7bf-4d68-bdc7-6568dc26d900-tls-certs\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.933495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.933416 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfwm\" (UniqueName: \"kubernetes.io/projected/9b32414f-a7bf-4d68-bdc7-6568dc26d900-kube-api-access-5lfwm\") pod \"kube-auth-proxy-578b8bf98-4lglr\" (UID: \"9b32414f-a7bf-4d68-bdc7-6568dc26d900\") " pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:42.989296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:42.989265 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" Oct 30 21:22:43.358499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:43.358431 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-578b8bf98-4lglr"] Oct 30 21:22:43.361660 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:22:43.361631 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b32414f_a7bf_4d68_bdc7_6568dc26d900.slice/crio-bb91cc25bb6f2127b0defeb96f32c368eb6ff079940ad6691ff6b19ff8ab6e4b WatchSource:0}: Error finding container bb91cc25bb6f2127b0defeb96f32c368eb6ff079940ad6691ff6b19ff8ab6e4b: Status 404 returned error can't find the container with id bb91cc25bb6f2127b0defeb96f32c368eb6ff079940ad6691ff6b19ff8ab6e4b Oct 30 21:22:43.513838 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:43.513807 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" event={"ID":"9b32414f-a7bf-4d68-bdc7-6568dc26d900","Type":"ContainerStarted","Data":"bb91cc25bb6f2127b0defeb96f32c368eb6ff079940ad6691ff6b19ff8ab6e4b"} Oct 30 21:22:45.195142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.195111 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:22:45.204147 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.204108 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.257243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.257208 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:22:45.258476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.258446 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:22:45.295155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.295118 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file\") pod \"limitador-limitador-74dfcfb894-jqxvc\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.295337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.295302 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt69\" (UniqueName: \"kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69\") pod \"limitador-limitador-74dfcfb894-jqxvc\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.307025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.306996 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:22:45.307025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.307012 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:22:45.331062 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.331030 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8m46b\"" Oct 30 21:22:45.396184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.396149 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt69\" (UniqueName: \"kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69\") pod \"limitador-limitador-74dfcfb894-jqxvc\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.396365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.396239 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file\") pod \"limitador-limitador-74dfcfb894-jqxvc\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.396872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.396849 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file\") pod \"limitador-limitador-74dfcfb894-jqxvc\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.430875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.430841 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:22:45.462852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.462749 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt69\" (UniqueName: \"kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69\") pod \"limitador-limitador-74dfcfb894-jqxvc\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.521762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.521485 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:45.904797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:45.904696 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:22:45.907550 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:22:45.907513 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b926bc_09be_4002_a90b_3da0b7ad1e28.slice/crio-de727a0d7f90df00762c444987ba7522f14feba3dd03b990f662147fc42295ab WatchSource:0}: Error finding container de727a0d7f90df00762c444987ba7522f14feba3dd03b990f662147fc42295ab: Status 404 returned error can't find the container with id de727a0d7f90df00762c444987ba7522f14feba3dd03b990f662147fc42295ab Oct 30 21:22:46.530419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:46.530328 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" event={"ID":"d1b926bc-09be-4002-a90b-3da0b7ad1e28","Type":"ContainerStarted","Data":"de727a0d7f90df00762c444987ba7522f14feba3dd03b990f662147fc42295ab"} Oct 30 21:22:46.531830 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:46.531802 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" event={"ID":"9b32414f-a7bf-4d68-bdc7-6568dc26d900","Type":"ContainerStarted","Data":"e171e4ec7e19da32cc5ab28bbdfe74227a0e0290a9ae4c0956f875e094ab565a"} Oct 30 21:22:49.469934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.469859 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-578b8bf98-4lglr" podStartSLOduration=4.564607474 podStartE2EDuration="7.469842005s" totalImagesPullingTime="2.905234531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:46.648497868 +0000 UTC m=+318.550361933" watchObservedRunningTime="2026-10-30 21:22:49.469842005 +0000 UTC m=+321.371706067" Oct 30 21:22:49.470542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.470525 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:22:49.473770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.473755 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:49.550815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.550731 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" event={"ID":"d1b926bc-09be-4002-a90b-3da0b7ad1e28","Type":"ContainerStarted","Data":"a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418"} Oct 30 21:22:49.550815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.550806 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:22:49.570655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.570629 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:22:49.637959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.637931 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9kc\" (UniqueName: \"kubernetes.io/projected/0a535427-8427-4cc8-a23b-030dcb993336-kube-api-access-4z9kc\") pod \"limitador-limitador-7c9b87d574-5lhdg\" (UID: \"0a535427-8427-4cc8-a23b-030dcb993336\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:49.638105 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.637979 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a535427-8427-4cc8-a23b-030dcb993336-config-file\") pod \"limitador-limitador-7c9b87d574-5lhdg\" (UID: \"0a535427-8427-4cc8-a23b-030dcb993336\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:49.700399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.700341 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" podStartSLOduration=1.32767217 podStartE2EDuration="4.700327003s" totalImagesPullingTime="3.372654833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:49.699138857 +0000 UTC m=+321.601002951" watchObservedRunningTime="2026-10-30 21:22:49.700327003 +0000 UTC m=+321.602191062" Oct 30 21:22:49.739032 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.738993 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9kc\" (UniqueName: \"kubernetes.io/projected/0a535427-8427-4cc8-a23b-030dcb993336-kube-api-access-4z9kc\") pod \"limitador-limitador-7c9b87d574-5lhdg\" (UID: \"0a535427-8427-4cc8-a23b-030dcb993336\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:49.739183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.739036 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a535427-8427-4cc8-a23b-030dcb993336-config-file\") pod \"limitador-limitador-7c9b87d574-5lhdg\" (UID: \"0a535427-8427-4cc8-a23b-030dcb993336\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:49.739669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.739652 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a535427-8427-4cc8-a23b-030dcb993336-config-file\") pod \"limitador-limitador-7c9b87d574-5lhdg\" (UID: \"0a535427-8427-4cc8-a23b-030dcb993336\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:49.778987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.778957 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:22:49.800567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:49.800539 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9kc\" (UniqueName: \"kubernetes.io/projected/0a535427-8427-4cc8-a23b-030dcb993336-kube-api-access-4z9kc\") pod \"limitador-limitador-7c9b87d574-5lhdg\" (UID: \"0a535427-8427-4cc8-a23b-030dcb993336\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:50.085871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:50.085842 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:50.453057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:50.452968 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:22:50.455662 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:22:50.455635 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a535427_8427_4cc8_a23b_030dcb993336.slice/crio-a5de382f5ea1004933a29edf195c5fca966625fd79fab4d5802a90fea46fc8eb WatchSource:0}: Error finding container a5de382f5ea1004933a29edf195c5fca966625fd79fab4d5802a90fea46fc8eb: Status 404 returned error can't find the container with id a5de382f5ea1004933a29edf195c5fca966625fd79fab4d5802a90fea46fc8eb Oct 30 21:22:50.556818 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:50.556781 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" event={"ID":"0a535427-8427-4cc8-a23b-030dcb993336","Type":"ContainerStarted","Data":"2edcf63d659427f264dd14a3caef0cc56299ca4dc2679f504887b181759ddd68"} Oct 30 21:22:50.557300 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:50.556838 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" event={"ID":"0a535427-8427-4cc8-a23b-030dcb993336","Type":"ContainerStarted","Data":"a5de382f5ea1004933a29edf195c5fca966625fd79fab4d5802a90fea46fc8eb"} Oct 30 21:22:50.557300 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:50.556928 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:22:50.672451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:22:50.672396 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" podStartSLOduration=1.67237918 podStartE2EDuration="1.67237918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.670179425 +0000 UTC m=+322.572043491" watchObservedRunningTime="2026-10-30 21:22:50.67237918 +0000 UTC m=+322.574243242" Oct 30 21:23:00.558584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:00.558526 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:23:01.562837 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:01.562806 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-5lhdg" Oct 30 21:23:01.731274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:01.726634 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:23:01.731274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:01.726975 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" podUID="d1b926bc-09be-4002-a90b-3da0b7ad1e28" containerName="limitador" containerID="cri-o://a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418" gracePeriod=30 Oct 30 21:23:02.278460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.278434 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:23:02.350101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.350019 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69\" (UniqueName: \"kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69\") pod \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " Oct 30 21:23:02.350101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.350095 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file\" (UniqueName: \"kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file\") pod \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\" (UID: \"d1b926bc-09be-4002-a90b-3da0b7ad1e28\") " Oct 30 21:23:02.350419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.350395 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file" pod "d1b926bc-09be-4002-a90b-3da0b7ad1e28" (UID: "d1b926bc-09be-4002-a90b-3da0b7ad1e28"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:02.352375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.352355 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69" pod "d1b926bc-09be-4002-a90b-3da0b7ad1e28" (UID: "d1b926bc-09be-4002-a90b-3da0b7ad1e28"). InnerVolumeSpecName "kube-api-access-qkt69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:02.450875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.450847 5914 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1b926bc-09be-4002-a90b-3da0b7ad1e28-config-file\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:02.451055 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.450873 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkt69\" (UniqueName: \"kubernetes.io/projected/d1b926bc-09be-4002-a90b-3da0b7ad1e28-kube-api-access-qkt69\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:02.608822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.608739 5914 generic.go:356] "Generic (PLEG): container finished" podID="d1b926bc-09be-4002-a90b-3da0b7ad1e28" containerID="a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418" exitCode=0 Oct 30 21:23:02.608822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.608780 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" event={"ID":"d1b926bc-09be-4002-a90b-3da0b7ad1e28","Type":"ContainerDied","Data":"a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418"} Oct 30 21:23:02.608822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.608796 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" Oct 30 21:23:02.609363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.608833 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jqxvc" event={"ID":"d1b926bc-09be-4002-a90b-3da0b7ad1e28","Type":"ContainerDied","Data":"de727a0d7f90df00762c444987ba7522f14feba3dd03b990f662147fc42295ab"} Oct 30 21:23:02.609363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.608858 5914 scope.go:122] "RemoveContainer" containerID="a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418" Oct 30 21:23:02.619504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.619483 5914 scope.go:122] "RemoveContainer" containerID="a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418" Oct 30 21:23:02.619804 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:23:02.619784 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418\": container with ID starting with a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418 not found: ID does not exist" containerID="a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418" Oct 30 21:23:02.619844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.619816 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418"} err="failed to get container status \"a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418\": rpc error: code = NotFound desc = could not find container \"a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418\": container with ID starting with a3485ce7ff5a78faabf7552b5713fdcbbb6f532e61985b56d047f054ea7a3418 not found: ID does not exist" Oct 30 21:23:02.772464 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.772427 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:23:02.821813 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:02.821779 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jqxvc"] Oct 30 21:23:04.215150 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:04.215116 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b926bc-09be-4002-a90b-3da0b7ad1e28" path="/var/lib/kubelet/pods/d1b926bc-09be-4002-a90b-3da0b7ad1e28/volumes" Oct 30 21:23:08.640431 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.640397 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn"] Oct 30 21:23:08.640802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.640779 5914 cpu_manager.go:414] "containerMap: removing container" podUID="d1b926bc-09be-4002-a90b-3da0b7ad1e28" containerName="limitador" Oct 30 21:23:08.640802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.640790 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b926bc-09be-4002-a90b-3da0b7ad1e28" containerName="limitador" Oct 30 21:23:08.640871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.640839 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b926bc-09be-4002-a90b-3da0b7ad1e28" containerName="limitador" Oct 30 21:23:08.643898 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.643867 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.696399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.696362 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:23:08.696399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.696388 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:23:08.700961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.700933 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78568b14-83eb-409f-ac90-5908ca352923-data\") pod \"postgres-7cdbf6dc44-xgzdn\" (UID: \"78568b14-83eb-409f-ac90-5908ca352923\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.701121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.700972 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnjw\" (UniqueName: \"kubernetes.io/projected/78568b14-83eb-409f-ac90-5908ca352923-kube-api-access-phnjw\") pod \"postgres-7cdbf6dc44-xgzdn\" (UID: \"78568b14-83eb-409f-ac90-5908ca352923\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.745907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.745846 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-stzln\"" Oct 30 21:23:08.745907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.745855 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 21:23:08.746798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.746777 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn"] Oct 30 21:23:08.801551 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.801515 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78568b14-83eb-409f-ac90-5908ca352923-data\") pod \"postgres-7cdbf6dc44-xgzdn\" (UID: \"78568b14-83eb-409f-ac90-5908ca352923\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.801747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.801560 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnjw\" (UniqueName: \"kubernetes.io/projected/78568b14-83eb-409f-ac90-5908ca352923-kube-api-access-phnjw\") pod \"postgres-7cdbf6dc44-xgzdn\" (UID: \"78568b14-83eb-409f-ac90-5908ca352923\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.802006 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.801985 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/78568b14-83eb-409f-ac90-5908ca352923-data\") pod \"postgres-7cdbf6dc44-xgzdn\" (UID: \"78568b14-83eb-409f-ac90-5908ca352923\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.858638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.858609 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnjw\" (UniqueName: \"kubernetes.io/projected/78568b14-83eb-409f-ac90-5908ca352923-kube-api-access-phnjw\") pod \"postgres-7cdbf6dc44-xgzdn\" (UID: \"78568b14-83eb-409f-ac90-5908ca352923\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:08.956526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:08.956441 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:09.333143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:09.333062 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn"] Oct 30 21:23:09.336385 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:23:09.336353 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78568b14_83eb_409f_ac90_5908ca352923.slice/crio-20f1a01c032b32c6ba260e52c99acd2ad6ac919e7cdd9aeaf535144fdfb4585e WatchSource:0}: Error finding container 20f1a01c032b32c6ba260e52c99acd2ad6ac919e7cdd9aeaf535144fdfb4585e: Status 404 returned error can't find the container with id 20f1a01c032b32c6ba260e52c99acd2ad6ac919e7cdd9aeaf535144fdfb4585e Oct 30 21:23:09.641637 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:09.641603 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" event={"ID":"78568b14-83eb-409f-ac90-5908ca352923","Type":"ContainerStarted","Data":"20f1a01c032b32c6ba260e52c99acd2ad6ac919e7cdd9aeaf535144fdfb4585e"} Oct 30 21:23:12.355898 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:23:12.355812 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:23:16.676046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:16.676006 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" event={"ID":"78568b14-83eb-409f-ac90-5908ca352923","Type":"ContainerStarted","Data":"480e282914c375a6005a257a1ad1ded4b65d560dffea7418779b85086bc73fd1"} Oct 30 21:23:16.676472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:16.676128 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:16.789716 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:16.789664 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" podStartSLOduration=1.9030869940000001 podStartE2EDuration="8.789650475s" totalImagesPullingTime="6.886563481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:16.789292638 +0000 UTC m=+348.691156702" watchObservedRunningTime="2026-10-30 21:23:16.789650475 +0000 UTC m=+348.691514541" Oct 30 21:23:22.710254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:22.710221 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-xgzdn" Oct 30 21:23:39.993211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:39.993130 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdpq4"] Oct 30 21:23:39.996458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:39.996406 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.049949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.049914 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ffhp5\"" Oct 30 21:23:40.078292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.078253 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmjr\" (UniqueName: \"kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.078482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.078304 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.078482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.078455 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.099198 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.099173 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:23:40.099491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.099477 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:23:40.099543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.099477 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:23:40.099675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.099658 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:23:40.099772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.099755 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:23:40.100742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.100724 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdpq4"] Oct 30 21:23:40.179420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.179390 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmjr\" (UniqueName: \"kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.179593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.179435 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.179593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.179499 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.182103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.182077 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.182233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.182178 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.238965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.238928 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmjr\" (UniqueName: \"kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr\") pod \"maas-controller-579cf7c6d-rdpq4\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.310014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.309923 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:40.678561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.678529 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdpq4"] Oct 30 21:23:40.681571 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:23:40.681541 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbb718d_1aff_45e0_bbe2_30fe7289f376.slice/crio-051ec534419dfb64bcbde96e998b64d650227b6670e0e14c45140cc625fc2230 WatchSource:0}: Error finding container 051ec534419dfb64bcbde96e998b64d650227b6670e0e14c45140cc625fc2230: Status 404 returned error can't find the container with id 051ec534419dfb64bcbde96e998b64d650227b6670e0e14c45140cc625fc2230 Oct 30 21:23:40.778650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:40.778618 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" event={"ID":"acbb718d-1aff-45e0-bbe2-30fe7289f376","Type":"ContainerStarted","Data":"051ec534419dfb64bcbde96e998b64d650227b6670e0e14c45140cc625fc2230"} Oct 30 21:23:42.351095 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:23:42.351053 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:23:44.800096 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:44.800055 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" event={"ID":"acbb718d-1aff-45e0-bbe2-30fe7289f376","Type":"ContainerStarted","Data":"e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54"} Oct 30 21:23:44.800509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:44.800319 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:44.920041 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:44.919989 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" podStartSLOduration=2.862849699 podStartE2EDuration="5.919972973s" totalImagesPullingTime="3.057123274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:44.918354591 +0000 UTC m=+376.820218652" watchObservedRunningTime="2026-10-30 21:23:44.919972973 +0000 UTC m=+376.821837037" Oct 30 21:23:55.806860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:55.806827 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:55.909986 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:55.909954 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg"] Oct 30 21:23:55.925695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:55.925669 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:55.995573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:55.995537 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg"] Oct 30 21:23:56.023165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023132 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.023165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023163 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.023384 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023202 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.023384 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023309 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6hlv\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.023384 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023349 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.023384 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023365 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.023523 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.023431 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.107795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.107700 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdpq4"] Oct 30 21:23:56.108243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.108211 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" podUID="acbb718d-1aff-45e0-bbe2-30fe7289f376" containerName="manager" containerID="cri-o://e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54" gracePeriod=30 Oct 30 21:23:56.124021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.123987 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.124178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.124129 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6hlv\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.124244 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.124204 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.124244 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.124231 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.124346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.124263 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.124346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.124298 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.124346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.124320 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.127730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.125660 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.127730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.127477 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.128685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.128653 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.136673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.136645 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.137504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.137479 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.183299 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.183266 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.183426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.183315 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6hlv\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-v6qfg\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.238840 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.238809 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.350643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.350617 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:56.427957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.427921 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs\") pod \"acbb718d-1aff-45e0-bbe2-30fe7289f376\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " Oct 30 21:23:56.428159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.427985 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert\") pod \"acbb718d-1aff-45e0-bbe2-30fe7289f376\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " Oct 30 21:23:56.428159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.428024 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr\" (UniqueName: \"kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr\") pod \"acbb718d-1aff-45e0-bbe2-30fe7289f376\" (UID: \"acbb718d-1aff-45e0-bbe2-30fe7289f376\") " Oct 30 21:23:56.430308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.430273 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs" pod "acbb718d-1aff-45e0-bbe2-30fe7289f376" (UID: "acbb718d-1aff-45e0-bbe2-30fe7289f376"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:56.430407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.430344 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert" pod "acbb718d-1aff-45e0-bbe2-30fe7289f376" (UID: "acbb718d-1aff-45e0-bbe2-30fe7289f376"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:56.430407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.430348 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr" pod "acbb718d-1aff-45e0-bbe2-30fe7289f376" (UID: "acbb718d-1aff-45e0-bbe2-30fe7289f376"). InnerVolumeSpecName "kube-api-access-wzmjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:56.529419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.529386 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:56.529419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.529413 5914 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/acbb718d-1aff-45e0-bbe2-30fe7289f376-webhook-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:56.529419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.529424 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzmjr\" (UniqueName: \"kubernetes.io/projected/acbb718d-1aff-45e0-bbe2-30fe7289f376-kube-api-access-wzmjr\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:23:56.671072 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.670988 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg"] Oct 30 21:23:56.676361 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:23:56.676320 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737311ce_3c3a_4cd9_a954_a889768bdb67.slice/crio-69e345c1ab9d01b07dd7bf6e0bc29e05aa2f85d5ab25576606d457ad2252e375 WatchSource:0}: Error finding container 69e345c1ab9d01b07dd7bf6e0bc29e05aa2f85d5ab25576606d457ad2252e375: Status 404 returned error can't find the container with id 69e345c1ab9d01b07dd7bf6e0bc29e05aa2f85d5ab25576606d457ad2252e375 Oct 30 21:23:56.678785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.678750 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:23:56.678874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.678824 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:23:56.855485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.855450 5914 generic.go:356] "Generic (PLEG): container finished" podID="acbb718d-1aff-45e0-bbe2-30fe7289f376" containerID="e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54" exitCode=0 Oct 30 21:23:56.855995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.855500 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" Oct 30 21:23:56.855995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.855534 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" event={"ID":"acbb718d-1aff-45e0-bbe2-30fe7289f376","Type":"ContainerDied","Data":"e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54"} Oct 30 21:23:56.855995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.855607 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-rdpq4" event={"ID":"acbb718d-1aff-45e0-bbe2-30fe7289f376","Type":"ContainerDied","Data":"051ec534419dfb64bcbde96e998b64d650227b6670e0e14c45140cc625fc2230"} Oct 30 21:23:56.855995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.855640 5914 scope.go:122] "RemoveContainer" containerID="e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54" Oct 30 21:23:56.857458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.857410 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" event={"ID":"737311ce-3c3a-4cd9-a954-a889768bdb67","Type":"ContainerStarted","Data":"a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7"} Oct 30 21:23:56.857590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.857474 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" event={"ID":"737311ce-3c3a-4cd9-a954-a889768bdb67","Type":"ContainerStarted","Data":"69e345c1ab9d01b07dd7bf6e0bc29e05aa2f85d5ab25576606d457ad2252e375"} Oct 30 21:23:56.857590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.857505 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:23:56.864429 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.864415 5914 scope.go:122] "RemoveContainer" containerID="e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54" Oct 30 21:23:56.864694 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:23:56.864678 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54\": container with ID starting with e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54 not found: ID does not exist" containerID="e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54" Oct 30 21:23:56.864728 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:56.864704 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54"} err="failed to get container status \"e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54\": rpc error: code = NotFound desc = could not find container \"e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54\": container with ID starting with e71aa40a35171d44630a5929f11a44c6253442d5f5ba226b1754c9890bec8a54 not found: ID does not exist" Oct 30 21:23:57.019457 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:57.019364 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdpq4"] Oct 30 21:23:57.023336 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:57.023310 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-rdpq4"] Oct 30 21:23:57.146690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:57.146638 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" podStartSLOduration=2.146623093 podStartE2EDuration="2.146623093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:57.14500688 +0000 UTC m=+389.046870946" watchObservedRunningTime="2026-10-30 21:23:57.146623093 +0000 UTC m=+389.048487157" Oct 30 21:23:57.866994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:57.866963 5914 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-v6qfg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:23:57.867473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:57.867014 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" podUID="737311ce-3c3a-4cd9-a954-a889768bdb67" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:23:58.214802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:23:58.214771 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbb718d-1aff-45e0-bbe2-30fe7289f376" path="/var/lib/kubelet/pods/acbb718d-1aff-45e0-bbe2-30fe7289f376/volumes" Oct 30 21:24:00.866402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:00.866372 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 21:24:01.108937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.108905 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp"] Oct 30 21:24:01.109135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.109112 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerName="discovery" containerID="cri-o://46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8" gracePeriod=30 Oct 30 21:24:01.362718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.362696 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:24:01.475794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475715 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.475989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475817 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.475989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475857 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.475989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475905 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.475989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475926 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.475989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475950 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.475989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.475969 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs\") pod \"a00ef032-c2c0-4f88-9c57-775f194c49d1\" (UID: \"a00ef032-c2c0-4f88-9c57-775f194c49d1\") " Oct 30 21:24:01.476330 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.476304 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:01.478692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.478658 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:01.478692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.478681 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:01.479180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.478767 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:01.479180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.478806 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "kube-api-access-tvv2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:01.479180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.478832 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:01.479180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.478976 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs" pod "a00ef032-c2c0-4f88-9c57-775f194c49d1" (UID: "a00ef032-c2c0-4f88-9c57-775f194c49d1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:01.577545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577514 5914 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-cacerts\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.577545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577541 5914 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-dns-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.577545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577551 5914 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-kubeconfig\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.577763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577560 5914 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a00ef032-c2c0-4f88-9c57-775f194c49d1-local-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.577763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577569 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvv2v\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-kube-api-access-tvv2v\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.577763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577579 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.577763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.577587 5914 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a00ef032-c2c0-4f88-9c57-775f194c49d1-istio-csr-ca-configmap\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:01.882370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.882337 5914 generic.go:356] "Generic (PLEG): container finished" podID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerID="46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8" exitCode=0 Oct 30 21:24:01.882760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.882385 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" Oct 30 21:24:01.882760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.882422 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" event={"ID":"a00ef032-c2c0-4f88-9c57-775f194c49d1","Type":"ContainerDied","Data":"46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8"} Oct 30 21:24:01.882760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.882479 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp" event={"ID":"a00ef032-c2c0-4f88-9c57-775f194c49d1","Type":"ContainerDied","Data":"8ec8eb513e9e866f1df440bf2ecb59ad57d29405b4e98effca60bebf0e08300b"} Oct 30 21:24:01.882760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.882500 5914 scope.go:122] "RemoveContainer" containerID="46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8" Oct 30 21:24:01.892232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.892215 5914 scope.go:122] "RemoveContainer" containerID="46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8" Oct 30 21:24:01.892518 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:24:01.892499 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8\": container with ID starting with 46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8 not found: ID does not exist" containerID="46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8" Oct 30 21:24:01.892599 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.892532 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8"} err="failed to get container status \"46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8\": rpc error: code = NotFound desc = could not find container \"46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8\": container with ID starting with 46e13078905428b831e4ef08d414c58d27474d745ec3dbbe6df553112acca7d8 not found: ID does not exist" Oct 30 21:24:01.916677 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.916648 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rmhhv"] Oct 30 21:24:01.917032 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.917019 5914 cpu_manager.go:414] "containerMap: removing container" podUID="acbb718d-1aff-45e0-bbe2-30fe7289f376" containerName="manager" Oct 30 21:24:01.917086 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.917034 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbb718d-1aff-45e0-bbe2-30fe7289f376" containerName="manager" Oct 30 21:24:01.917086 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.917046 5914 cpu_manager.go:414] "containerMap: removing container" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerName="discovery" Oct 30 21:24:01.917086 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.917051 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerName="discovery" Oct 30 21:24:01.917195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.917122 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbb718d-1aff-45e0-bbe2-30fe7289f376" containerName="manager" Oct 30 21:24:01.917195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.917131 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" containerName="discovery" Oct 30 21:24:01.920123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.920105 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:01.972201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.972171 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:24:01.972201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.972184 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6pc86\"" Oct 30 21:24:01.972201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.972196 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:24:01.972411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.972284 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:24:01.977175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.977149 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rmhhv"] Oct 30 21:24:01.981576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.981554 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwtq\" (UniqueName: \"kubernetes.io/projected/957309ba-a518-4e31-b235-850d1d0a87bd-kube-api-access-5gwtq\") pod \"kserve-controller-manager-776cc66d47-rmhhv\" (UID: \"957309ba-a518-4e31-b235-850d1d0a87bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:01.981690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:01.981619 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/957309ba-a518-4e31-b235-850d1d0a87bd-cert\") pod \"kserve-controller-manager-776cc66d47-rmhhv\" (UID: \"957309ba-a518-4e31-b235-850d1d0a87bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:02.082573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.082538 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwtq\" (UniqueName: \"kubernetes.io/projected/957309ba-a518-4e31-b235-850d1d0a87bd-kube-api-access-5gwtq\") pod \"kserve-controller-manager-776cc66d47-rmhhv\" (UID: \"957309ba-a518-4e31-b235-850d1d0a87bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:02.082770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.082596 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/957309ba-a518-4e31-b235-850d1d0a87bd-cert\") pod \"kserve-controller-manager-776cc66d47-rmhhv\" (UID: \"957309ba-a518-4e31-b235-850d1d0a87bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:02.085172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.085150 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/957309ba-a518-4e31-b235-850d1d0a87bd-cert\") pod \"kserve-controller-manager-776cc66d47-rmhhv\" (UID: \"957309ba-a518-4e31-b235-850d1d0a87bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:02.143932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.143844 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwtq\" (UniqueName: \"kubernetes.io/projected/957309ba-a518-4e31-b235-850d1d0a87bd-kube-api-access-5gwtq\") pod \"kserve-controller-manager-776cc66d47-rmhhv\" (UID: \"957309ba-a518-4e31-b235-850d1d0a87bd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:02.158472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.158431 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp"] Oct 30 21:24:02.162163 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.162138 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jlgnp"] Oct 30 21:24:02.215177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.215149 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00ef032-c2c0-4f88-9c57-775f194c49d1" path="/var/lib/kubelet/pods/a00ef032-c2c0-4f88-9c57-775f194c49d1/volumes" Oct 30 21:24:02.232046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.232016 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:02.244871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.244845 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-25psm"] Oct 30 21:24:02.248460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.248433 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.287594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.286419 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-cert\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.287594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.286563 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-metrics-certs\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.287594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.286621 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkct\" (UniqueName: \"kubernetes.io/projected/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-kube-api-access-vdkct\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.301651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.300644 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 21:24:02.301651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.300644 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 21:24:02.301651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.300971 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5s292\"" Oct 30 21:24:02.305652 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.305619 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-25psm"] Oct 30 21:24:02.387615 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.387578 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-metrics-certs\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.387794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.387646 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkct\" (UniqueName: \"kubernetes.io/projected/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-kube-api-access-vdkct\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.387794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.387729 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-cert\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.390396 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.390373 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-cert\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.390520 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.390500 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-metrics-certs\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.444786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.444712 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkct\" (UniqueName: \"kubernetes.io/projected/ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2-kube-api-access-vdkct\") pod \"odh-model-controller-769875db56-25psm\" (UID: \"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2\") " pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.569771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.569731 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:02.603623 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.603600 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rmhhv"] Oct 30 21:24:02.606984 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:24:02.606952 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957309ba_a518_4e31_b235_850d1d0a87bd.slice/crio-6dfa902e1d4d43ca48cae5450ef7f1be9db42562bed4f94991398fe7de89e017 WatchSource:0}: Error finding container 6dfa902e1d4d43ca48cae5450ef7f1be9db42562bed4f94991398fe7de89e017: Status 404 returned error can't find the container with id 6dfa902e1d4d43ca48cae5450ef7f1be9db42562bed4f94991398fe7de89e017 Oct 30 21:24:02.887732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.887696 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" event={"ID":"957309ba-a518-4e31-b235-850d1d0a87bd","Type":"ContainerStarted","Data":"6dfa902e1d4d43ca48cae5450ef7f1be9db42562bed4f94991398fe7de89e017"} Oct 30 21:24:02.940440 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:02.940417 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-25psm"] Oct 30 21:24:02.942975 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:24:02.942948 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4f2d63_5bcf_4303_93b8_7279ba0f7cf2.slice/crio-cc1ce6dffe5232c0d9a64bdaeee1a5ec946ab930fc32815a95f56b24fae5c5f7 WatchSource:0}: Error finding container cc1ce6dffe5232c0d9a64bdaeee1a5ec946ab930fc32815a95f56b24fae5c5f7: Status 404 returned error can't find the container with id cc1ce6dffe5232c0d9a64bdaeee1a5ec946ab930fc32815a95f56b24fae5c5f7 Oct 30 21:24:03.895259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:03.895222 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-25psm" event={"ID":"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2","Type":"ContainerStarted","Data":"cc1ce6dffe5232c0d9a64bdaeee1a5ec946ab930fc32815a95f56b24fae5c5f7"} Oct 30 21:24:05.908611 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:05.908561 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" event={"ID":"957309ba-a518-4e31-b235-850d1d0a87bd","Type":"ContainerStarted","Data":"91956f24e15a0a9d0c33496af127c7370f7db5520fcbc0ceb049dfa22c2e0bba"} Oct 30 21:24:05.909111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:05.908755 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:06.915119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:06.915080 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-25psm" event={"ID":"ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2","Type":"ContainerStarted","Data":"e23eaf79e0680ebacce725a7bbe300b6f2a8a803c95446fff9ea23f31a9b6722"} Oct 30 21:24:06.915605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:06.915395 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:07.031629 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:07.031562 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-25psm" podStartSLOduration=1.946039389 podStartE2EDuration="5.031542149s" totalImagesPullingTime="3.08550276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:07.031033598 +0000 UTC m=+398.932897674" watchObservedRunningTime="2026-10-30 21:24:07.031542149 +0000 UTC m=+398.933406213" Oct 30 21:24:07.031828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:07.031769 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" podStartSLOduration=3.185234762 podStartE2EDuration="6.031760439s" totalImagesPullingTime="2.846525677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:06.023925205 +0000 UTC m=+397.925789268" watchObservedRunningTime="2026-10-30 21:24:07.031760439 +0000 UTC m=+398.933624506" Oct 30 21:24:12.344285 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:24:12.344246 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:24:17.921975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:17.921941 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-25psm" Oct 30 21:24:19.458297 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.458261 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd"] Oct 30 21:24:19.462437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.462420 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.554871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.554832 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.555060 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.554899 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.555060 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.555021 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4jq8\" (UniqueName: \"kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.555136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.555059 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.567183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.567148 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:24:19.567183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.567149 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:24:19.567369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.567223 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bk7w4\"" Oct 30 21:24:19.568313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.568292 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd"] Oct 30 21:24:19.655744 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.655706 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.655950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.655767 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.655950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.655804 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.655950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.655862 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4jq8\" (UniqueName: \"kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.656557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.656492 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.658591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.658543 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.658591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.658561 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.712801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.712725 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4jq8\" (UniqueName: \"kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8\") pod \"maas-api-6d669c7fcd-2x8kd\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:19.774199 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:19.774164 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:20.141922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:20.141867 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd"] Oct 30 21:24:20.144950 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:24:20.144918 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdfc6a4_64ac_4a7d_8fec_7d1b79f713fc.slice/crio-6c71861e98c3cb5c2f31d13546ba0b4f044a3a01c81a7d99cd2d33687aee5f1c WatchSource:0}: Error finding container 6c71861e98c3cb5c2f31d13546ba0b4f044a3a01c81a7d99cd2d33687aee5f1c: Status 404 returned error can't find the container with id 6c71861e98c3cb5c2f31d13546ba0b4f044a3a01c81a7d99cd2d33687aee5f1c Oct 30 21:24:20.977852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:20.977818 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" event={"ID":"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc","Type":"ContainerStarted","Data":"6c71861e98c3cb5c2f31d13546ba0b4f044a3a01c81a7d99cd2d33687aee5f1c"} Oct 30 21:24:22.988830 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:22.988798 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" event={"ID":"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc","Type":"ContainerStarted","Data":"a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4"} Oct 30 21:24:22.989258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:22.988924 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:23.104413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:23.104362 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" podStartSLOduration=2.186464822 podStartE2EDuration="4.104347739s" totalImagesPullingTime="1.917882917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:23.103924176 +0000 UTC m=+415.005788244" watchObservedRunningTime="2026-10-30 21:24:23.104347739 +0000 UTC m=+415.006211803" Oct 30 21:24:24.825291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:24.825258 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:24:28.998483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:28.998452 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:29.993346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:29.993311 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-4zll5"] Oct 30 21:24:29.998141 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:29.998111 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.058273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.058240 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-4zll5"] Oct 30 21:24:30.152648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.152615 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.152648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.152650 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.152861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.152671 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf65l\" (UniqueName: \"kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.152861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.152814 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.253462 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.253366 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.253625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.253520 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.253625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.253552 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.253625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.253575 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf65l\" (UniqueName: \"kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.254118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.254087 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.256105 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.256083 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.256410 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.256391 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.313653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.313613 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf65l\" (UniqueName: \"kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l\") pod \"maas-api-745b76d875-4zll5\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.611496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.611412 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:30.983183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:30.983066 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-4zll5"] Oct 30 21:24:30.985799 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:24:30.985770 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03b79e3_577c_4a1f_a90c_aa48807550b6.slice/crio-3f3378b7833a5e2c13e257a6f2d54f2c830f6f5dc24073918002c8b8094689af WatchSource:0}: Error finding container 3f3378b7833a5e2c13e257a6f2d54f2c830f6f5dc24073918002c8b8094689af: Status 404 returned error can't find the container with id 3f3378b7833a5e2c13e257a6f2d54f2c830f6f5dc24073918002c8b8094689af Oct 30 21:24:31.022629 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:31.022584 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" event={"ID":"d03b79e3-577c-4a1f-a90c-aa48807550b6","Type":"ContainerStarted","Data":"3f3378b7833a5e2c13e257a6f2d54f2c830f6f5dc24073918002c8b8094689af"} Oct 30 21:24:32.029111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:32.029074 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" event={"ID":"d03b79e3-577c-4a1f-a90c-aa48807550b6","Type":"ContainerStarted","Data":"4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c"} Oct 30 21:24:32.029633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:32.029213 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:32.144964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:32.144907 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" podStartSLOduration=2.339218667 podStartE2EDuration="3.144869166s" totalImagesPullingTime="805.650499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:32.143869301 +0000 UTC m=+424.045733370" watchObservedRunningTime="2026-10-30 21:24:32.144869166 +0000 UTC m=+424.046733238" Oct 30 21:24:36.920434 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:36.920401 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rmhhv" Oct 30 21:24:38.035577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:38.035538 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": dial tcp 10.134.0.32:8443: connect: connection refused" Oct 30 21:24:42.075188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:42.075156 5914 generic.go:356] "Generic (PLEG): container finished" podID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerID="4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c" exitCode=1 Oct 30 21:24:42.075584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:42.075231 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" event={"ID":"d03b79e3-577c-4a1f-a90c-aa48807550b6","Type":"ContainerDied","Data":"4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c"} Oct 30 21:24:42.075584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:42.075576 5914 scope.go:122] "RemoveContainer" containerID="4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c" Oct 30 21:24:42.344437 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:24:42.344353 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:24:43.081485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:43.081441 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" event={"ID":"d03b79e3-577c-4a1f-a90c-aa48807550b6","Type":"ContainerStarted","Data":"0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b"} Oct 30 21:24:43.082016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:43.081764 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:49.090638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.090603 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 21:24:49.256437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.256408 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd"] Oct 30 21:24:49.256640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.256612 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" podUID="1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" containerName="maas-api" containerID="cri-o://a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4" gracePeriod=30 Oct 30 21:24:49.498018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.497994 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:49.630801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.630728 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8\" (UniqueName: \"kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8\") pod \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " Oct 30 21:24:49.630801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.630783 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls\") pod \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " Oct 30 21:24:49.631017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.630845 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca\") pod \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " Oct 30 21:24:49.631017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.630893 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs\") pod \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\" (UID: \"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc\") " Oct 30 21:24:49.631394 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.631350 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca" pod "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" (UID: "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:49.633385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.633355 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls" pod "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" (UID: "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:49.633522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.633504 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs" pod "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" (UID: "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:49.633596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.633582 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8" pod "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" (UID: "1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc"). InnerVolumeSpecName "kube-api-access-d4jq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:49.731841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.731805 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.731841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.731836 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.731841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.731847 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4jq8\" (UniqueName: \"kubernetes.io/projected/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-kube-api-access-d4jq8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.732088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:49.731856 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:24:50.113688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.113600 5914 generic.go:356] "Generic (PLEG): container finished" podID="1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" containerID="a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4" exitCode=0 Oct 30 21:24:50.113688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.113647 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" Oct 30 21:24:50.114148 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.113696 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" event={"ID":"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc","Type":"ContainerDied","Data":"a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4"} Oct 30 21:24:50.114148 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.113747 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd" event={"ID":"1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc","Type":"ContainerDied","Data":"6c71861e98c3cb5c2f31d13546ba0b4f044a3a01c81a7d99cd2d33687aee5f1c"} Oct 30 21:24:50.114148 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.113770 5914 scope.go:122] "RemoveContainer" containerID="a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4" Oct 30 21:24:50.123193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.123176 5914 scope.go:122] "RemoveContainer" containerID="a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4" Oct 30 21:24:50.123457 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:24:50.123439 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4\": container with ID starting with a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4 not found: ID does not exist" containerID="a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4" Oct 30 21:24:50.123510 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.123467 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4"} err="failed to get container status \"a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4\": rpc error: code = NotFound desc = could not find container \"a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4\": container with ID starting with a515ad4d3596c646f030123f86eacf6b37ec68dc2a45aa6ec32686a24d4d39d4 not found: ID does not exist" Oct 30 21:24:50.277025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.276991 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd"] Oct 30 21:24:50.279158 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:50.279135 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2x8kd"] Oct 30 21:24:52.215243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:24:52.215208 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" path="/var/lib/kubelet/pods/1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc/volumes" Oct 30 21:25:12.344696 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:25:12.344605 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:25:42.345836 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:25:42.345795 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:26:05.927472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:05.927436 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4qch8"] Oct 30 21:26:05.928115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:05.928094 5914 cpu_manager.go:414] "containerMap: removing container" podUID="1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" containerName="maas-api" Oct 30 21:26:05.928201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:05.928118 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" containerName="maas-api" Oct 30 21:26:05.928309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:05.928295 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdfc6a4-64ac-4a7d-8fec-7d1b79f713fc" containerName="maas-api" Oct 30 21:26:05.931543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:05.931519 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.005434 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.005401 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-home\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.005606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.005465 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10913186-88c2-40e0-ba50-a2c7f26c64ad-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.005606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.005516 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2rg\" (UniqueName: \"kubernetes.io/projected/10913186-88c2-40e0-ba50-a2c7f26c64ad-kube-api-access-xf2rg\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.005687 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.005633 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.005687 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.005663 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.005687 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.005683 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.036470 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.036442 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 21:26:06.036806 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.036788 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tv6dw\"" Oct 30 21:26:06.036911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.036786 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:26:06.036911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.036786 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:26:06.038024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.038002 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4qch8"] Oct 30 21:26:06.106867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.106834 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10913186-88c2-40e0-ba50-a2c7f26c64ad-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.106867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.106867 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2rg\" (UniqueName: \"kubernetes.io/projected/10913186-88c2-40e0-ba50-a2c7f26c64ad-kube-api-access-xf2rg\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.106929 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.106957 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.106982 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.107067 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-home\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.107454 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.107480 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-home\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.107845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.107523 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.109275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.109255 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10913186-88c2-40e0-ba50-a2c7f26c64ad-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.109633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.109610 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10913186-88c2-40e0-ba50-a2c7f26c64ad-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.165725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.165689 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2rg\" (UniqueName: \"kubernetes.io/projected/10913186-88c2-40e0-ba50-a2c7f26c64ad-kube-api-access-xf2rg\") pod \"echo-simulator-kserve-64b6f8f96c-4qch8\" (UID: \"10913186-88c2-40e0-ba50-a2c7f26c64ad\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.244667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.244590 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:06.615123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.615052 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4qch8"] Oct 30 21:26:06.618287 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:26:06.618259 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10913186_88c2_40e0_ba50_a2c7f26c64ad.slice/crio-454ad16c2d81769c95af2e17d445555b04adebe07d14cd4fc2279dc3008f1a2d WatchSource:0}: Error finding container 454ad16c2d81769c95af2e17d445555b04adebe07d14cd4fc2279dc3008f1a2d: Status 404 returned error can't find the container with id 454ad16c2d81769c95af2e17d445555b04adebe07d14cd4fc2279dc3008f1a2d Oct 30 21:26:06.619633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:06.619614 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:26:07.449420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:07.449389 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" event={"ID":"10913186-88c2-40e0-ba50-a2c7f26c64ad","Type":"ContainerStarted","Data":"454ad16c2d81769c95af2e17d445555b04adebe07d14cd4fc2279dc3008f1a2d"} Oct 30 21:26:09.373091 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:09.373060 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:26:09.460865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:09.460830 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" event={"ID":"10913186-88c2-40e0-ba50-a2c7f26c64ad","Type":"ContainerStarted","Data":"e0d2dc9db07663f1478619f903ccc3fb77e3aa5308ed214e014c346565c4711a"} Oct 30 21:26:09.580620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:09.580564 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" podStartSLOduration=2.115096593 podStartE2EDuration="4.580547499s" totalImagesPullingTime="2.465450906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:09.579488603 +0000 UTC m=+521.481352693" watchObservedRunningTime="2026-10-30 21:26:09.580547499 +0000 UTC m=+521.482411565" Oct 30 21:26:10.245593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:10.245556 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:12.345159 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:26:12.345117 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:26:16.245002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.244971 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:16.257446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.257416 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:16.502669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.502603 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4qch8" Oct 30 21:26:16.609876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.609844 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m"] Oct 30 21:26:16.613522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.613501 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.666100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.666070 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 21:26:16.672764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.672731 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m"] Oct 30 21:26:16.705475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.705437 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.705648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.705489 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a9ebae-7b84-4141-8410-4b0b9225c81f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.705648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.705559 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.705648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.705594 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twlln\" (UniqueName: \"kubernetes.io/projected/58a9ebae-7b84-4141-8410-4b0b9225c81f-kube-api-access-twlln\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.705770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.705675 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.705770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.705703 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806368 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twlln\" (UniqueName: \"kubernetes.io/projected/58a9ebae-7b84-4141-8410-4b0b9225c81f-kube-api-access-twlln\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806432 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806456 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806600 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806644 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a9ebae-7b84-4141-8410-4b0b9225c81f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806672 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.806907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.806861 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.807035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.807014 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.807076 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.807021 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.809211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.809185 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a9ebae-7b84-4141-8410-4b0b9225c81f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.809418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.809399 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a9ebae-7b84-4141-8410-4b0b9225c81f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.863476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.863445 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twlln\" (UniqueName: \"kubernetes.io/projected/58a9ebae-7b84-4141-8410-4b0b9225c81f-kube-api-access-twlln\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-p878m\" (UID: \"58a9ebae-7b84-4141-8410-4b0b9225c81f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:16.926208 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:16.926177 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:17.301307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:17.301176 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m"] Oct 30 21:26:17.303576 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:26:17.303539 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a9ebae_7b84_4141_8410_4b0b9225c81f.slice/crio-54bb89aec7dce34ef4023ad777b10c8aa9ba63e7e2e681270fb0d6a0d876e592 WatchSource:0}: Error finding container 54bb89aec7dce34ef4023ad777b10c8aa9ba63e7e2e681270fb0d6a0d876e592: Status 404 returned error can't find the container with id 54bb89aec7dce34ef4023ad777b10c8aa9ba63e7e2e681270fb0d6a0d876e592 Oct 30 21:26:17.497138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:17.497098 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" event={"ID":"58a9ebae-7b84-4141-8410-4b0b9225c81f","Type":"ContainerStarted","Data":"54bb89aec7dce34ef4023ad777b10c8aa9ba63e7e2e681270fb0d6a0d876e592"} Oct 30 21:26:19.087039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:19.086949 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:26:19.507786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:19.507751 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" event={"ID":"58a9ebae-7b84-4141-8410-4b0b9225c81f","Type":"ContainerStarted","Data":"204b374482b710808721cc1971614a6ec4748e31882f227eddc88b907e3d0f3f"} Oct 30 21:26:19.625275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:19.625223 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" podStartSLOduration=2.109931362 podStartE2EDuration="3.625208417s" totalImagesPullingTime="1.515277055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:19.624448059 +0000 UTC m=+531.526312124" watchObservedRunningTime="2026-10-30 21:26:19.625208417 +0000 UTC m=+531.527072492" Oct 30 21:26:19.926870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:19.926826 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:24.208764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.208728 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4"] Oct 30 21:26:24.212720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.212696 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.277631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.277602 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.277797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.277636 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsw6\" (UniqueName: \"kubernetes.io/projected/0902fa42-613e-4b60-a4f5-c6e0b0857801-kube-api-access-vrsw6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.277797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.277660 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.277987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.277960 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.278028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.278014 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.278061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.278046 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0902fa42-613e-4b60-a4f5-c6e0b0857801-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.317396 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.317364 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:26:24.318471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.318451 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4"] Oct 30 21:26:24.378818 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.378785 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.378818 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.378823 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.378840 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0902fa42-613e-4b60-a4f5-c6e0b0857801-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.378870 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.378923 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsw6\" (UniqueName: \"kubernetes.io/projected/0902fa42-613e-4b60-a4f5-c6e0b0857801-kube-api-access-vrsw6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.378954 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.379216 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.379343 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.379430 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.379371 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.381331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.381309 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0902fa42-613e-4b60-a4f5-c6e0b0857801-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.381566 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.381546 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0902fa42-613e-4b60-a4f5-c6e0b0857801-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.437291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.437261 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsw6\" (UniqueName: \"kubernetes.io/projected/0902fa42-613e-4b60-a4f5-c6e0b0857801-kube-api-access-vrsw6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kg4x4\" (UID: \"0902fa42-613e-4b60-a4f5-c6e0b0857801\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.525775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.525686 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:24.901539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:24.901512 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4"] Oct 30 21:26:24.904067 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:26:24.904037 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0902fa42_613e_4b60_a4f5_c6e0b0857801.slice/crio-7333c5d823813dd56587000a684d647684a9ef1b0edaf69a7699e7596391a399 WatchSource:0}: Error finding container 7333c5d823813dd56587000a684d647684a9ef1b0edaf69a7699e7596391a399: Status 404 returned error can't find the container with id 7333c5d823813dd56587000a684d647684a9ef1b0edaf69a7699e7596391a399 Oct 30 21:26:25.534218 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:25.534187 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" event={"ID":"0902fa42-613e-4b60-a4f5-c6e0b0857801","Type":"ContainerStarted","Data":"7333c5d823813dd56587000a684d647684a9ef1b0edaf69a7699e7596391a399"} Oct 30 21:26:26.540313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:26.540280 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" event={"ID":"0902fa42-613e-4b60-a4f5-c6e0b0857801","Type":"ContainerStarted","Data":"dee772865003b5f3b1f56408df90789da58720299df74a1f270db466606d3f08"} Oct 30 21:26:26.655332 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:26.655277 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" podStartSLOduration=2.05846281 podStartE2EDuration="2.655262065s" totalImagesPullingTime="596.799255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:26.654301231 +0000 UTC m=+538.556165308" watchObservedRunningTime="2026-10-30 21:26:26.655262065 +0000 UTC m=+538.557126129" Oct 30 21:26:26.927017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:26.926981 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:26.939951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:26.939924 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:27.526860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:27.526828 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:27.555876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:27.555848 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-p878m" Oct 30 21:26:30.352137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.352104 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc"] Oct 30 21:26:30.355719 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.355703 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.408104 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.408077 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x8rmh\"" Oct 30 21:26:30.440404 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440369 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440427 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440530 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440573 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440598 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440655 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5q4n\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440853 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440796 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.440853 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440825 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.441032 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.440860 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.458954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.458919 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc"] Oct 30 21:26:30.542288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542255 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542309 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542427 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542463 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542485 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q4n\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542568 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542593 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542615 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.542730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542664 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.543002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.542911 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.543061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.543007 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.543283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.543079 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.543283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.543087 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.543395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.543341 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.544910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.544866 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.545120 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.545099 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.599308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.599279 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q4n\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.599471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.599290 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjrxc\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.644629 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.644597 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b"] Oct 30 21:26:30.649372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.649352 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.668730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.668710 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:30.709030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.708977 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:26:30.710914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.710374 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b"] Oct 30 21:26:30.745253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.745220 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.745426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.745265 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.745426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.745292 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.745426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.745310 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3715d9-a970-425c-88df-4f3544ea45d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.745426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.745334 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54hp\" (UniqueName: \"kubernetes.io/projected/0e3715d9-a970-425c-88df-4f3544ea45d9-kube-api-access-l54hp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.745426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.745403 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846403 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846447 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846468 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3715d9-a970-425c-88df-4f3544ea45d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846493 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54hp\" (UniqueName: \"kubernetes.io/projected/0e3715d9-a970-425c-88df-4f3544ea45d9-kube-api-access-l54hp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846583 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846617 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846872 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.846986 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.846941 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.847022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.847005 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.848906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.848872 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e3715d9-a970-425c-88df-4f3544ea45d9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.849090 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.849075 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3715d9-a970-425c-88df-4f3544ea45d9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.906388 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.906325 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54hp\" (UniqueName: \"kubernetes.io/projected/0e3715d9-a970-425c-88df-4f3544ea45d9-kube-api-access-l54hp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8mp4b\" (UID: \"0e3715d9-a970-425c-88df-4f3544ea45d9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:30.963035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:30.963001 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:31.098034 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:31.098008 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc"] Oct 30 21:26:31.100069 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:26:31.100043 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1049aab_2a9b_4c8b_ac8b_83a2b7b0d24f.slice/crio-7569ddc6dd7ccaf5321e476a9b24e9f78950836255fac19b6db2885d3a26f0f1 WatchSource:0}: Error finding container 7569ddc6dd7ccaf5321e476a9b24e9f78950836255fac19b6db2885d3a26f0f1: Status 404 returned error can't find the container with id 7569ddc6dd7ccaf5321e476a9b24e9f78950836255fac19b6db2885d3a26f0f1 Oct 30 21:26:31.336310 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:31.336229 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b"] Oct 30 21:26:31.339318 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:26:31.339290 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3715d9_a970_425c_88df_4f3544ea45d9.slice/crio-51d4cccbae3d8076a2fc45537b137c35cccfc19f92843999d2a82b9c782764cc WatchSource:0}: Error finding container 51d4cccbae3d8076a2fc45537b137c35cccfc19f92843999d2a82b9c782764cc: Status 404 returned error can't find the container with id 51d4cccbae3d8076a2fc45537b137c35cccfc19f92843999d2a82b9c782764cc Oct 30 21:26:31.565305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:31.565273 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" event={"ID":"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f","Type":"ContainerStarted","Data":"7569ddc6dd7ccaf5321e476a9b24e9f78950836255fac19b6db2885d3a26f0f1"} Oct 30 21:26:31.566364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:31.566341 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" event={"ID":"0e3715d9-a970-425c-88df-4f3544ea45d9","Type":"ContainerStarted","Data":"51d4cccbae3d8076a2fc45537b137c35cccfc19f92843999d2a82b9c782764cc"} Oct 30 21:26:32.579261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:32.579221 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" event={"ID":"0e3715d9-a970-425c-88df-4f3544ea45d9","Type":"ContainerStarted","Data":"0438473133d6bba04ce4124d28fa263e05a142d13bbea403e947351f66b5c422"} Oct 30 21:26:32.694214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:32.694164 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" podStartSLOduration=2.114878516 podStartE2EDuration="2.694149563s" totalImagesPullingTime="579.271047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:32.693437879 +0000 UTC m=+544.595301945" watchObservedRunningTime="2026-10-30 21:26:32.694149563 +0000 UTC m=+544.596013627" Oct 30 21:26:32.964147 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:32.964107 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:34.526174 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:34.526137 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:34.544419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:34.544391 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:34.605177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:34.605144 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kg4x4" Oct 30 21:26:34.798523 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:34.798483 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:34.798633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:34.798579 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:34.798672 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:34.798627 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:26:35.256177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:35.256144 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:26:35.604763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:35.604673 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" event={"ID":"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f","Type":"ContainerStarted","Data":"3c4179e6699bc899d75a9261992dc70ee50178f9d51a1dc685912533de2113f4"} Oct 30 21:26:35.669976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:35.669943 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:35.671478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:35.671445 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:35.671612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:35.671494 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:35.722005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:35.721941 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podStartSLOduration=2.026618963 podStartE2EDuration="5.721922669s" totalImagesPullingTime="3.695303706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:35.721823361 +0000 UTC m=+547.623687428" watchObservedRunningTime="2026-10-30 21:26:35.721922669 +0000 UTC m=+547.623786734" Oct 30 21:26:36.669914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:36.669824 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:36.669914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:36.669874 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:37.669621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:37.669584 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:37.669811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:37.669645 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:38.669505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:38.669468 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:38.669929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:38.669522 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:39.669471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:39.669432 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:39.669650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:39.669499 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:40.669814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:40.669780 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:40.670227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:40.669840 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:40.963927 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:40.963820 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:40.977071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:40.977047 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:41.644766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:41.644740 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8mp4b" Oct 30 21:26:41.669588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:41.669554 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:26:41.669718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:41.669618 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:26:42.345862 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:26:42.345806 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:26:43.711079 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:43.711040 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjrxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:43.711534 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:43.711128 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:44.452721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:44.452683 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:44.453234 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:44.453211 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:44.454175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:44.454152 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:26:46.237602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:26:46.237567 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:27:00.595390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.595354 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59"] Oct 30 21:27:00.602421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.602397 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.690030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.689997 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59"] Oct 30 21:27:00.723405 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723372 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723571 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723447 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723571 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723506 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723571 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723547 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723571 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723590 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723771 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgfc\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723827 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.723924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.723869 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825295 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825336 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825392 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825418 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825440 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825482 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825507 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825618 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgfc\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825715 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.825827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825798 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.826022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825992 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.826081 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.825993 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.826131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.826113 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.826293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.826267 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.827911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.827860 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.828240 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.828219 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.883350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.883326 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.883875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.883853 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgfc\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc\") pod \"maas-default-gateway-openshift-default-7f694df8bc-rnc59\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:00.914717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:00.914689 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:01.336709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.336627 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59"] Oct 30 21:27:01.340082 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:27:01.340055 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178f0036_19d7_4e38_b37e_e60ad4fec147.slice/crio-c6bf310cff4b9c99f6e3d7d9980e2e8bd1c88c06347337c1a48429d78e1542c0 WatchSource:0}: Error finding container c6bf310cff4b9c99f6e3d7d9980e2e8bd1c88c06347337c1a48429d78e1542c0: Status 404 returned error can't find the container with id c6bf310cff4b9c99f6e3d7d9980e2e8bd1c88c06347337c1a48429d78e1542c0 Oct 30 21:27:01.342340 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.342302 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:01.342429 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.342385 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:01.342466 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.342431 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:01.725764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.725731 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" event={"ID":"178f0036-19d7-4e38-b37e-e60ad4fec147","Type":"ContainerStarted","Data":"f7a16265cb8eccb62fd3d23c587efdad4d21bf00c396a5e424ccd1b02abe96f2"} Oct 30 21:27:01.726304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.725782 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" event={"ID":"178f0036-19d7-4e38-b37e-e60ad4fec147","Type":"ContainerStarted","Data":"c6bf310cff4b9c99f6e3d7d9980e2e8bd1c88c06347337c1a48429d78e1542c0"} Oct 30 21:27:01.843485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.843436 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podStartSLOduration=1.843416843 podStartE2EDuration="1.843416843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:01.842798269 +0000 UTC m=+573.744662339" watchObservedRunningTime="2026-10-30 21:27:01.843416843 +0000 UTC m=+573.745280907" Oct 30 21:27:01.915468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:01.915420 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:02.917247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:02.917210 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:27:02.917659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:02.917267 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:27:03.915825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:03.915788 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:27:03.916028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:03.915846 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:27:04.916124 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:04.916086 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:27:04.916526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:04.916151 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:27:05.915602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:05.915548 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:27:05.915818 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:05.915617 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:27:06.915526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:06.915488 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:27:06.915951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:06.915558 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:27:07.915665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:07.915630 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 21:27:07.916084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:07.915685 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 21:27:08.890538 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:08.890484 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:27:09.916169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:09.916121 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:27:09.916645 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:09.916194 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:10.958064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:10.958020 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-rnc59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:27:10.958592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:10.958097 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:11.932451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:11.932422 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:11.932747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:11.932715 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:11.975936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:11.975863 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:27:12.346002 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:27:12.345916 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:27:27.923681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:27.923649 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fgz7d"] Oct 30 21:27:27.930834 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:27.930811 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:27.983695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:27.983662 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fgz7d"] Oct 30 21:27:28.016536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.016498 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e2cb45-b7df-4a03-8a39-883b748adfb5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fgz7d\" (UID: \"22e2cb45-b7df-4a03-8a39-883b748adfb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.016703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.016551 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwc2\" (UniqueName: \"kubernetes.io/projected/22e2cb45-b7df-4a03-8a39-883b748adfb5-kube-api-access-jpwc2\") pod \"payload-processing-55fbbbf9dc-fgz7d\" (UID: \"22e2cb45-b7df-4a03-8a39-883b748adfb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.031406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.031378 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:27:28.031573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.031554 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8rdsw\"" Oct 30 21:27:28.117931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.117895 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e2cb45-b7df-4a03-8a39-883b748adfb5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fgz7d\" (UID: \"22e2cb45-b7df-4a03-8a39-883b748adfb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.118115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.117980 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwc2\" (UniqueName: \"kubernetes.io/projected/22e2cb45-b7df-4a03-8a39-883b748adfb5-kube-api-access-jpwc2\") pod \"payload-processing-55fbbbf9dc-fgz7d\" (UID: \"22e2cb45-b7df-4a03-8a39-883b748adfb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.118589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.118564 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e2cb45-b7df-4a03-8a39-883b748adfb5-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fgz7d\" (UID: \"22e2cb45-b7df-4a03-8a39-883b748adfb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.175592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.175518 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwc2\" (UniqueName: \"kubernetes.io/projected/22e2cb45-b7df-4a03-8a39-883b748adfb5-kube-api-access-jpwc2\") pod \"payload-processing-55fbbbf9dc-fgz7d\" (UID: \"22e2cb45-b7df-4a03-8a39-883b748adfb5\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.295022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.294989 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8rdsw\"" Oct 30 21:27:28.295534 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.295514 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:27:28.297951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.297925 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:27:28.304417 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.304397 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:28.677726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.677663 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fgz7d"] Oct 30 21:27:28.680004 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:27:28.679978 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e2cb45_b7df_4a03_8a39_883b748adfb5.slice/crio-ed68dcef7e9dec94f83c41fae5f59b3e088007a179e7e20b8c002309619a840a WatchSource:0}: Error finding container ed68dcef7e9dec94f83c41fae5f59b3e088007a179e7e20b8c002309619a840a: Status 404 returned error can't find the container with id ed68dcef7e9dec94f83c41fae5f59b3e088007a179e7e20b8c002309619a840a Oct 30 21:27:28.853119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:28.853079 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" event={"ID":"22e2cb45-b7df-4a03-8a39-883b748adfb5","Type":"ContainerStarted","Data":"ed68dcef7e9dec94f83c41fae5f59b3e088007a179e7e20b8c002309619a840a"} Oct 30 21:27:30.827690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.827648 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j"] Oct 30 21:27:30.839268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.839235 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947569 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947629 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947736 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947788 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947831 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947920 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947943 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9dcx\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.947985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.947965 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:30.948095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:30.948010 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.033580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.033495 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j"] Oct 30 21:27:31.059532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059494 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.059769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059550 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.059769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059573 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.059769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059619 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.059769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059657 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.059769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059710 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.059769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059739 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9dcx\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.060177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059770 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.060177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.059817 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.060177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.060087 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.060714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.060608 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.061263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.061239 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.061416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.061392 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.061720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.061695 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.064136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.064084 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.066728 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.066680 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.118788 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.118712 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9dcx\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.120332 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.120306 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-2462j\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.157681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.157644 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:31.578787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.578757 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j"] Oct 30 21:27:31.581398 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:27:31.581373 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3a3936_5efb_4bfd_a45f_1f62cf689329.slice/crio-6a53f65013352e6c0075908b36b669881cd86ea33f23f79d7cae66e5c69a4cd0 WatchSource:0}: Error finding container 6a53f65013352e6c0075908b36b669881cd86ea33f23f79d7cae66e5c69a4cd0: Status 404 returned error can't find the container with id 6a53f65013352e6c0075908b36b669881cd86ea33f23f79d7cae66e5c69a4cd0 Oct 30 21:27:31.583625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.583594 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.583702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.583665 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.583702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.583694 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:27:31.873593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.873550 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" event={"ID":"aa3a3936-5efb-4bfd-a45f-1f62cf689329","Type":"ContainerStarted","Data":"d523295b8c19fa78ba74ec27235ac8245266b5d5ee04f47bf5df4b6540a0c58b"} Oct 30 21:27:31.874105 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.873622 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" event={"ID":"aa3a3936-5efb-4bfd-a45f-1f62cf689329","Type":"ContainerStarted","Data":"6a53f65013352e6c0075908b36b669881cd86ea33f23f79d7cae66e5c69a4cd0"} Oct 30 21:27:31.875337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.875245 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" event={"ID":"22e2cb45-b7df-4a03-8a39-883b748adfb5","Type":"ContainerStarted","Data":"a4c235812a20d3281e7e3020ac171310c87caf75f72d74f8d755c68bea8c34ea"} Oct 30 21:27:31.875483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:31.875397 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:27:32.029992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:32.029911 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podStartSLOduration=2.029868548 podStartE2EDuration="2.029868548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:32.028603377 +0000 UTC m=+603.930467441" watchObservedRunningTime="2026-10-30 21:27:32.029868548 +0000 UTC m=+603.931732620" Oct 30 21:27:32.143548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:32.143433 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" podStartSLOduration=2.620156354 podStartE2EDuration="5.143415557s" totalImagesPullingTime="2.523259203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:32.142199097 +0000 UTC m=+604.044063161" watchObservedRunningTime="2026-10-30 21:27:32.143415557 +0000 UTC m=+604.045279653" Oct 30 21:27:32.158507 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:32.158465 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:32.159931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:32.159905 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:32.160072 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:32.159955 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:33.158915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:33.158851 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:33.159331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:33.158942 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:34.158100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:34.158066 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:34.158299 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:34.158118 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:35.159089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:35.159053 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:35.159491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:35.159122 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:36.158641 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:36.158609 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:36.158791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:36.158662 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:37.158791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:37.158753 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:37.159338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:37.158819 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:38.158319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:38.158284 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-2462j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 21:27:38.158495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:38.158349 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 21:27:39.905799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:39.905766 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:39.906799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:39.906771 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:39.907584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:39.907563 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:27:42.344515 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:27:42.344472 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:27:42.904080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:27:42.904054 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fgz7d" Oct 30 21:28:12.344948 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:28:12.344854 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:28:42.345029 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:28:42.344995 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:28:48.163786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.163744 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp"] Oct 30 21:28:48.168134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.168094 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.221389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.221360 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9796f1-openshift-default-dockercfg-bzdq6\"" Oct 30 21:28:48.224504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.224472 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp"] Oct 30 21:28:48.291737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.291692 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.291973 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.291776 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.291973 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.291818 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.291973 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.291854 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.291973 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.291953 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9td29\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.292213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.291989 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.292213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.292085 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.292213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.292136 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.292213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.292189 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393292 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393342 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393395 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9td29\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393429 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393491 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393533 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393580 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393619 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393671 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.393967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.393768 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.394094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.394069 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.394159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.394070 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.394232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.394213 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.394454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.394429 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.396003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.395981 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.396122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.396093 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.442160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.442096 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr"] Oct 30 21:28:48.446326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.446306 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.450076 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.450050 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9td29\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.450393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.450375 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token\") pod \"e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.502352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.502320 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e4dadd-openshift-default-dockercfg-vmhcw\"" Oct 30 21:28:48.503814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.503787 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr"] Oct 30 21:28:48.524416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.524390 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:48.602000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.601964 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602118 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602183 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602278 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602237 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhml\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602337 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602431 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602415 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602450 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602538 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602509 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.602632 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703396 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703441 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703472 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703500 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703526 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhml\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703545 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703576 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703592 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.703777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703636 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.704545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.703874 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.704545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.704015 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.704545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.704061 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.704545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.704116 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.704713 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.704550 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.706719 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.706679 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.706827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.706809 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.763918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.763856 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.764100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.764081 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhml\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.775022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.774998 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:48.957486 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.957461 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp"] Oct 30 21:28:48.959923 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:28:48.959891 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bad27b_019f_4b10_b8d7_1a68b898992f.slice/crio-1a66552be5b007b34993276b611783fcdf6b44eb4b50a973ee39aad356f045f6 WatchSource:0}: Error finding container 1a66552be5b007b34993276b611783fcdf6b44eb4b50a973ee39aad356f045f6: Status 404 returned error can't find the container with id 1a66552be5b007b34993276b611783fcdf6b44eb4b50a973ee39aad356f045f6 Oct 30 21:28:48.962132 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.962101 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:48.962225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.962170 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:48.962225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:48.962199 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.160449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.160427 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr"] Oct 30 21:28:49.163073 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:28:49.163044 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79bd64b_0a3b_45bd_94bd_f5426fbf94c4.slice/crio-6f38af54c9d7e685e5fdfea3fcce05fa1e589d7d68bac8df839d3df40e5cda27 WatchSource:0}: Error finding container 6f38af54c9d7e685e5fdfea3fcce05fa1e589d7d68bac8df839d3df40e5cda27: Status 404 returned error can't find the container with id 6f38af54c9d7e685e5fdfea3fcce05fa1e589d7d68bac8df839d3df40e5cda27 Oct 30 21:28:49.165726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.165698 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.166064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.165766 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.166064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.165796 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:49.271686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.271658 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" event={"ID":"60bad27b-019f-4b10-b8d7-1a68b898992f","Type":"ContainerStarted","Data":"99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7"} Oct 30 21:28:49.271907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.271715 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" event={"ID":"60bad27b-019f-4b10-b8d7-1a68b898992f","Type":"ContainerStarted","Data":"1a66552be5b007b34993276b611783fcdf6b44eb4b50a973ee39aad356f045f6"} Oct 30 21:28:49.273359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.273332 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" event={"ID":"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4","Type":"ContainerStarted","Data":"9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a"} Oct 30 21:28:49.273495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.273380 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" event={"ID":"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4","Type":"ContainerStarted","Data":"6f38af54c9d7e685e5fdfea3fcce05fa1e589d7d68bac8df839d3df40e5cda27"} Oct 30 21:28:49.391256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.391199 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podStartSLOduration=1.391182178 podStartE2EDuration="1.391182178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:49.391075626 +0000 UTC m=+681.292939691" watchObservedRunningTime="2026-10-30 21:28:49.391182178 +0000 UTC m=+681.293046247" Oct 30 21:28:49.532839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.530455 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:49.776003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:49.775966 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:50.530451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:50.530421 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:50.715559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:50.715505 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podStartSLOduration=2.715487111 podStartE2EDuration="2.715487111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:49.514136682 +0000 UTC m=+681.416000747" watchObservedRunningTime="2026-10-30 21:28:50.715487111 +0000 UTC m=+682.617351177" Oct 30 21:28:50.780276 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:50.780245 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:51.286154 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:51.286114 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:51.286346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:51.286177 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:51.286910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:51.286856 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:28:51.287035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:51.286943 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:28:57.275741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.275705 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94"] Oct 30 21:28:57.279798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.279780 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.310663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.310632 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brt59\" (UniqueName: \"kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59\") pod \"payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.310895 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.310860 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.333978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.333948 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-021907\"" Oct 30 21:28:57.391603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.391556 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94"] Oct 30 21:28:57.391805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.391752 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-021907-dockercfg-29f4g\"" Oct 30 21:28:57.412141 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.412105 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brt59\" (UniqueName: \"kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59\") pod \"payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.412402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.412275 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.413113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.413087 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.424093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.424065 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd"] Oct 30 21:28:57.427871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.427854 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.469857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.469823 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brt59\" (UniqueName: \"kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59\") pod \"payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.485868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.485835 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd"] Oct 30 21:28:57.513137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.513101 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5ls\" (UniqueName: \"kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls\") pod \"payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.513312 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.513238 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.593409 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.593324 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:57.613874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.613846 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.614053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.614026 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5ls\" (UniqueName: \"kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls\") pod \"payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.614542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.614524 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.677316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.676365 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5ls\" (UniqueName: \"kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls\") pod \"payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.741529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.741498 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:28:57.768288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.768248 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq"] Oct 30 21:28:57.772500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.772476 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821012 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821101 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821137 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2zt\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821186 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821216 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821255 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821312 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821358 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.823143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.821396 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.854094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.853122 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq"] Oct 30 21:28:57.854094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.854082 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1b5b5f-openshift-default-dockercfg-kzsbh\"" Oct 30 21:28:57.922112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922082 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922168 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922200 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2zt\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922244 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922274 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922310 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922368 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922413 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922451 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922612 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.922955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922930 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.923049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.922967 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.923049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.923026 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.923207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.923184 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.924726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.924704 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.924822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.924788 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.979678 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.979363 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94"] Oct 30 21:28:57.982156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.982123 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.982710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:57.982435 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2zt\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:57.982710 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:28:57.982500 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac64985c_d072_4a7b_b1a9_18733974a80b.slice/crio-7511b4ee49e2ebee1c11c503dcfd2128e3ff065c9e2fb4e682503dbe74abed53 WatchSource:0}: Error finding container 7511b4ee49e2ebee1c11c503dcfd2128e3ff065c9e2fb4e682503dbe74abed53: Status 404 returned error can't find the container with id 7511b4ee49e2ebee1c11c503dcfd2128e3ff065c9e2fb4e682503dbe74abed53 Oct 30 21:28:58.087089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.087061 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:28:58.148797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.129136 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd"] Oct 30 21:28:58.323353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.323303 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" event={"ID":"fb8eb883-8773-4f50-8f67-b3621be81239","Type":"ContainerStarted","Data":"09c842cea57aa9458d4395771ca82c923f660447ce8dcc5e6020bcf01ec6f067"} Oct 30 21:28:58.325957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.325928 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp"] Oct 30 21:28:58.333050 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.333022 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" event={"ID":"ac64985c-d072-4a7b-b1a9-18733974a80b","Type":"ContainerStarted","Data":"3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860"} Oct 30 21:28:58.333153 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.333064 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" event={"ID":"ac64985c-d072-4a7b-b1a9-18733974a80b","Type":"ContainerStarted","Data":"7511b4ee49e2ebee1c11c503dcfd2128e3ff065c9e2fb4e682503dbe74abed53"} Oct 30 21:28:58.333153 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.333132 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:28:58.333232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.333170 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.386253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.386225 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-9796f1-serving-cert\"" Oct 30 21:28:58.388829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.388801 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp"] Oct 30 21:28:58.442665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.442565 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" podStartSLOduration=1.442550657 podStartE2EDuration="1.442550657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:58.441159196 +0000 UTC m=+690.343023461" watchObservedRunningTime="2026-10-30 21:28:58.442550657 +0000 UTC m=+690.344414722" Oct 30 21:28:58.458304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.458267 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjrh\" (UniqueName: \"kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.458470 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.458319 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.458522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.458496 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.458685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.458666 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.506431 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.506400 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq"] Oct 30 21:28:58.510194 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:28:58.510169 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37547fb6_b031_4e65_ab9a_4bfa41ea3056.slice/crio-ecdb81f940bf936759d0b45f5beb06015c6f5fb4c13f01fddeae4695d0f78af6 WatchSource:0}: Error finding container ecdb81f940bf936759d0b45f5beb06015c6f5fb4c13f01fddeae4695d0f78af6: Status 404 returned error can't find the container with id ecdb81f940bf936759d0b45f5beb06015c6f5fb4c13f01fddeae4695d0f78af6 Oct 30 21:28:58.512814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.512783 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:58.512915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.512847 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:58.512915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.512909 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:28:58.559710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.559682 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.559843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.559785 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjrh\" (UniqueName: \"kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.559843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.559816 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.560053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.559849 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.560053 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:28:58.559864 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-serving-cert: secret "maas-api-e2e-worker-w3-9796f1-serving-cert" not found Oct 30 21:28:58.560053 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:28:58.559957 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls podName:78c32271-7a11-4546-92bf-2706c934fa12 nodeName:}" failed. No retries permitted until 2026-10-30 21:28:59.059936991 +0000 UTC m=+690.961801051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls") pod "maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" (UID: "78c32271-7a11-4546-92bf-2706c934fa12") : secret "maas-api-e2e-worker-w3-9796f1-serving-cert" not found Oct 30 21:28:58.560673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.560648 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.562814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.562787 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:58.619536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:58.619500 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjrh\" (UniqueName: \"kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:59.068716 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.068674 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:59.071269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.071238 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:59.246843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.246811 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:28:59.336584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.336484 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" event={"ID":"37547fb6-b031-4e65-ab9a-4bfa41ea3056","Type":"ContainerStarted","Data":"107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986"} Oct 30 21:28:59.336584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.336560 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" event={"ID":"37547fb6-b031-4e65-ab9a-4bfa41ea3056","Type":"ContainerStarted","Data":"ecdb81f940bf936759d0b45f5beb06015c6f5fb4c13f01fddeae4695d0f78af6"} Oct 30 21:28:59.338258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.338222 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" event={"ID":"fb8eb883-8773-4f50-8f67-b3621be81239","Type":"ContainerStarted","Data":"d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a"} Oct 30 21:28:59.454195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.454140 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podStartSLOduration=2.4541241879999998 podStartE2EDuration="2.454124188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:59.452389951 +0000 UTC m=+691.354254016" watchObservedRunningTime="2026-10-30 21:28:59.454124188 +0000 UTC m=+691.355988321" Oct 30 21:28:59.568108 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.567967 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" podStartSLOduration=2.567947245 podStartE2EDuration="2.567947245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:59.56578285 +0000 UTC m=+691.467646916" watchObservedRunningTime="2026-10-30 21:28:59.567947245 +0000 UTC m=+691.469811310" Oct 30 21:28:59.623362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:28:59.623153 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp"] Oct 30 21:28:59.629730 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:28:59.629671 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c32271_7a11_4546_92bf_2706c934fa12.slice/crio-e052cf4754f042dbc6490188b971240faafc59460ce0e750e1ed3ac11ad3abbb WatchSource:0}: Error finding container e052cf4754f042dbc6490188b971240faafc59460ce0e750e1ed3ac11ad3abbb: Status 404 returned error can't find the container with id e052cf4754f042dbc6490188b971240faafc59460ce0e750e1ed3ac11ad3abbb Oct 30 21:29:00.088184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:00.088100 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:29:00.093481 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:00.093453 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:29:00.345110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:00.345013 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" event={"ID":"78c32271-7a11-4546-92bf-2706c934fa12","Type":"ContainerStarted","Data":"e052cf4754f042dbc6490188b971240faafc59460ce0e750e1ed3ac11ad3abbb"} Oct 30 21:29:00.345601 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:00.345328 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:29:00.345857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:00.345836 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:29:00.346400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:00.346378 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:29:01.352113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:01.352063 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" event={"ID":"78c32271-7a11-4546-92bf-2706c934fa12","Type":"ContainerStarted","Data":"ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0"} Oct 30 21:29:01.352617 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:01.352544 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:29:01.479268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:01.479209 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" podStartSLOduration=2.6811568340000003 podStartE2EDuration="3.479193592s" totalImagesPullingTime="798.036758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:01.477837343 +0000 UTC m=+693.379701433" watchObservedRunningTime="2026-10-30 21:29:01.479193592 +0000 UTC m=+693.381057657" Oct 30 21:29:03.549122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.549083 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l"] Oct 30 21:29:03.553415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.553391 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.619558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.619520 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e4dadd-serving-cert\"" Oct 30 21:29:03.620706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.620678 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l"] Oct 30 21:29:03.660204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.660168 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.660404 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.660286 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwwzm\" (UniqueName: \"kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.660404 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.660338 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.660404 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.660375 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.677833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.677799 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:29:03.761978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.761933 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.762167 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:03.762038 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-serving-cert: secret "maas-api-e2e-worker-w1-e4dadd-serving-cert" not found Oct 30 21:29:03.762167 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:03.762113 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls podName:dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:04.262092419 +0000 UTC m=+696.163956485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls") pod "maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" (UID: "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66") : secret "maas-api-e2e-worker-w1-e4dadd-serving-cert" not found Oct 30 21:29:03.762348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.762040 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwwzm\" (UniqueName: \"kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.762348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.762303 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.762456 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.762350 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.763046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.763022 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.764805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.764781 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:03.827936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:03.821870 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwwzm\" (UniqueName: \"kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:04.268576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:04.268536 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:04.305985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:04.305953 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:04.467667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:04.467629 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:04.917162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:04.917065 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l"] Oct 30 21:29:04.919945 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:29:04.919916 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce0c4c5_9e4c_4d0c_869c_f8105f3b6a66.slice/crio-5ccb05d1d5fc7924e61aa7a49a91ec1df476924385cb6b2689d04144dfb74358 WatchSource:0}: Error finding container 5ccb05d1d5fc7924e61aa7a49a91ec1df476924385cb6b2689d04144dfb74358: Status 404 returned error can't find the container with id 5ccb05d1d5fc7924e61aa7a49a91ec1df476924385cb6b2689d04144dfb74358 Oct 30 21:29:05.398514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:05.398464 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" event={"ID":"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66","Type":"ContainerStarted","Data":"5ccb05d1d5fc7924e61aa7a49a91ec1df476924385cb6b2689d04144dfb74358"} Oct 30 21:29:06.413998 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:06.413953 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" event={"ID":"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66","Type":"ContainerStarted","Data":"3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8"} Oct 30 21:29:06.414409 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:06.414106 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:06.533708 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:06.533656 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" podStartSLOduration=2.64538597 podStartE2EDuration="3.533641393s" totalImagesPullingTime="888.255423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:06.532766075 +0000 UTC m=+698.434630151" watchObservedRunningTime="2026-10-30 21:29:06.533641393 +0000 UTC m=+698.435505457" Oct 30 21:29:07.363574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:07.363547 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:29:08.809983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.809937 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k"] Oct 30 21:29:08.814257 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.814231 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:08.917077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.917053 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-07fa8e-serving-cert\"" Oct 30 21:29:08.918923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.918868 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k"] Oct 30 21:29:08.953439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.953410 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnssh\" (UniqueName: \"kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:08.953618 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.953600 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:08.953670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.953632 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:08.953670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:08.953656 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.065972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.065832 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnssh\" (UniqueName: \"kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.066453 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.066425 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.066657 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.066641 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.067812 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.067790 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.068553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.067732 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.068759 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:09.066561 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-serving-cert: secret "maas-api-e2e-worker-w4-07fa8e-serving-cert" not found Oct 30 21:29:09.068942 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:09.068928 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls podName:7abed670-4638-47a4-acf3-d3eb0b2d7c58 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:09.56890401 +0000 UTC m=+701.470768076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls") pod "maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" (UID: "7abed670-4638-47a4-acf3-d3eb0b2d7c58") : secret "maas-api-e2e-worker-w4-07fa8e-serving-cert" not found Oct 30 21:29:09.072321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.072273 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.122938 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.122907 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnssh\" (UniqueName: \"kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.573898 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.573843 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.576443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.576414 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\") pod \"maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:09.727411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:09.727378 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:10.101607 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:10.101577 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k"] Oct 30 21:29:10.103558 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:29:10.103530 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abed670_4638_47a4_acf3_d3eb0b2d7c58.slice/crio-eede64f306c1c02d21fcb48756371329cd0e53655e12f849c7db6a79ace8e7c2 WatchSource:0}: Error finding container eede64f306c1c02d21fcb48756371329cd0e53655e12f849c7db6a79ace8e7c2: Status 404 returned error can't find the container with id eede64f306c1c02d21fcb48756371329cd0e53655e12f849c7db6a79ace8e7c2 Oct 30 21:29:10.348552 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:10.348520 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:29:10.435564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:10.435523 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" event={"ID":"7abed670-4638-47a4-acf3-d3eb0b2d7c58","Type":"ContainerStarted","Data":"eede64f306c1c02d21fcb48756371329cd0e53655e12f849c7db6a79ace8e7c2"} Oct 30 21:29:11.355913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:11.355808 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:29:11.442641 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:11.442609 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" event={"ID":"7abed670-4638-47a4-acf3-d3eb0b2d7c58","Type":"ContainerStarted","Data":"0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f"} Oct 30 21:29:11.442831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:11.442675 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:11.606039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:11.605925 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" podStartSLOduration=2.745252345 podStartE2EDuration="3.605906664s" totalImagesPullingTime="860.654319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:11.603751367 +0000 UTC m=+703.505615433" watchObservedRunningTime="2026-10-30 21:29:11.605906664 +0000 UTC m=+703.507770730" Oct 30 21:29:12.208905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:12.205585 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:29:12.395548 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:12.395509 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:29:12.425050 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:12.425024 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:29:13.468525 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:13.468448 5914 generic.go:356] "Generic (PLEG): container finished" podID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerID="99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7" exitCode=0 Oct 30 21:29:13.469088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:13.468540 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" event={"ID":"60bad27b-019f-4b10-b8d7-1a68b898992f","Type":"ContainerDied","Data":"99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7"} Oct 30 21:29:13.469088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:13.468952 5914 scope.go:122] "RemoveContainer" containerID="99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7" Oct 30 21:29:13.470487 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:13.470453 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:13.470604 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:13.470542 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:13.470604 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:13.470590 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:14.484913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:14.482713 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" event={"ID":"60bad27b-019f-4b10-b8d7-1a68b898992f","Type":"ContainerStarted","Data":"04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894"} Oct 30 21:29:14.528907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:14.528278 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:29:14.529117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:14.528948 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:29:14.529920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:14.529099 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:29:15.526034 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:15.525997 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:29:15.526654 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:15.526061 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:29:16.525258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:16.525224 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:29:16.525583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:16.525536 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:29:17.462438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:17.460250 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 21:29:17.524794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:17.524758 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:29:17.525147 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:17.525109 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:29:18.525290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:18.525257 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:29:18.525742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:18.525318 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:29:19.524979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:19.524941 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:29:19.525188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:19.524996 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:29:20.529162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:20.529129 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:29:20.529635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:20.529384 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:29:20.530151 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:20.530132 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:29:21.862062 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:21.860713 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:35804->10.134.0.42:15021: read: connection reset by peer" start-of-body= Oct 30 21:29:21.862062 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:21.860801 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": read tcp 10.134.0.2:35804->10.134.0.42:15021: read: connection reset by peer" Oct 30 21:29:22.067164 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.067080 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:29:22.565431 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.565403 5914 generic.go:356] "Generic (PLEG): container finished" podID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerID="9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a" exitCode=0 Oct 30 21:29:22.565624 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.565466 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" event={"ID":"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4","Type":"ContainerDied","Data":"9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a"} Oct 30 21:29:22.565852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.565839 5914 scope.go:122] "RemoveContainer" containerID="9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a" Oct 30 21:29:22.567235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.567209 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:22.567305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.567277 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:22.567352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:22.567308 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:29:23.600162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.600112 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" event={"ID":"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4","Type":"ContainerStarted","Data":"7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a"} Oct 30 21:29:23.771385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.771346 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb"] Oct 30 21:29:23.793396 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.793358 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:29:23.793689 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.793659 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:29:23.798144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.798106 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:29:23.798288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.798246 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:23.813107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.813075 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:23.813247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.813117 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8dz\" (UniqueName: \"kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz\") pod \"payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:23.857979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.852332 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1b5b5f-dockercfg-5w9h4\"" Oct 30 21:29:23.857979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.853257 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1b5b5f\"" Oct 30 21:29:23.857979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.857031 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb"] Oct 30 21:29:23.919172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.918557 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:23.919172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.918611 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8dz\" (UniqueName: \"kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz\") pod \"payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:23.919528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:23.919501 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:24.001724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.001692 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8dz\" (UniqueName: \"kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz\") pod \"payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:24.112276 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.112184 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:24.505329 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.505297 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb"] Oct 30 21:29:24.506562 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:29:24.506509 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457f7406_0588_4389_9731_550c7ae5f988.slice/crio-84990d09fb394c2ae83f05c95bc6e4244dd6d03cb9c1fbe085a64d50ba0040d0 WatchSource:0}: Error finding container 84990d09fb394c2ae83f05c95bc6e4244dd6d03cb9c1fbe085a64d50ba0040d0: Status 404 returned error can't find the container with id 84990d09fb394c2ae83f05c95bc6e4244dd6d03cb9c1fbe085a64d50ba0040d0 Oct 30 21:29:24.608030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.607992 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" event={"ID":"457f7406-0588-4389-9731-550c7ae5f988","Type":"ContainerStarted","Data":"36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0"} Oct 30 21:29:24.608461 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.608056 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" event={"ID":"457f7406-0588-4389-9731-550c7ae5f988","Type":"ContainerStarted","Data":"84990d09fb394c2ae83f05c95bc6e4244dd6d03cb9c1fbe085a64d50ba0040d0"} Oct 30 21:29:24.723333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.723265 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" podStartSLOduration=1.723251097 podStartE2EDuration="1.723251097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:24.721433512 +0000 UTC m=+716.623297580" watchObservedRunningTime="2026-10-30 21:29:24.723251097 +0000 UTC m=+716.625115161" Oct 30 21:29:24.797959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.797364 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:29:24.797959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:24.797425 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:29:25.612870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:25.612837 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:25.785928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:25.782289 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:29:25.785928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:25.782353 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:29:26.780321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:26.780283 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:29:26.780935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:26.780901 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:29:27.776326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:27.776288 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:29:27.776607 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:27.776579 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:29:28.792965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:28.792933 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:29:28.793444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:28.793254 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:29:28.794556 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:28.794525 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:29:32.372273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:32.372236 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:29:34.835524 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.835480 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw"] Oct 30 21:29:34.840964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.840942 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:34.897183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.897148 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw"] Oct 30 21:29:34.944039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.943999 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-89c0fd-serving-cert\"" Oct 30 21:29:34.991251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.991216 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfsg\" (UniqueName: \"kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:34.991420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.991258 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:34.991420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.991360 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:34.991539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:34.991520 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.093247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.093161 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.093247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.093222 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfsg\" (UniqueName: \"kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.093247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.093246 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.093521 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.093276 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.093521 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:35.093381 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-serving-cert: secret "maas-api-e2e-worker-w6-89c0fd-serving-cert" not found Oct 30 21:29:35.093521 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:35.093450 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls podName:5142aeba-c339-44d3-aa6a-4c3636f46950 nodeName:}" failed. No retries permitted until 2026-10-30 21:29:35.593429618 +0000 UTC m=+727.495293681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls") pod "maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" (UID: "5142aeba-c339-44d3-aa6a-4c3636f46950") : secret "maas-api-e2e-worker-w6-89c0fd-serving-cert" not found Oct 30 21:29:35.094071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.094046 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.095678 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.095656 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.152439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.152402 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfsg\" (UniqueName: \"kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.598242 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.598204 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.600739 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.600709 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\") pod \"maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:35.754274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:35.754230 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:36.129991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:36.129963 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw"] Oct 30 21:29:36.132225 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:29:36.132198 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5142aeba_c339_44d3_aa6a_4c3636f46950.slice/crio-607e60c81f0df3eddcf61b1bedb89f4dee9f8c4abcadedf162daf3aa5e80b2f7 WatchSource:0}: Error finding container 607e60c81f0df3eddcf61b1bedb89f4dee9f8c4abcadedf162daf3aa5e80b2f7: Status 404 returned error can't find the container with id 607e60c81f0df3eddcf61b1bedb89f4dee9f8c4abcadedf162daf3aa5e80b2f7 Oct 30 21:29:36.623657 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:36.623631 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:29:36.683462 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:36.683429 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" event={"ID":"5142aeba-c339-44d3-aa6a-4c3636f46950","Type":"ContainerStarted","Data":"607e60c81f0df3eddcf61b1bedb89f4dee9f8c4abcadedf162daf3aa5e80b2f7"} Oct 30 21:29:37.690166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:37.690128 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" event={"ID":"5142aeba-c339-44d3-aa6a-4c3636f46950","Type":"ContainerStarted","Data":"cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea"} Oct 30 21:29:37.690555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:37.690246 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:37.825325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:37.824592 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" podStartSLOduration=2.994035027 podStartE2EDuration="3.824574053s" totalImagesPullingTime="830.539026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:37.820188403 +0000 UTC m=+729.722052469" watchObservedRunningTime="2026-10-30 21:29:37.824574053 +0000 UTC m=+729.726438118" Oct 30 21:29:42.387433 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:29:42.386320 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:29:42.562433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:42.562387 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:29:43.701048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:43.701020 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:29:51.865979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:29:51.865939 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:30:10.495588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.495545 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:30:10.911563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.911524 5914 generic.go:356] "Generic (PLEG): container finished" podID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerID="107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986" exitCode=0 Oct 30 21:30:10.911752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.911663 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" event={"ID":"37547fb6-b031-4e65-ab9a-4bfa41ea3056","Type":"ContainerDied","Data":"107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986"} Oct 30 21:30:10.912190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.912165 5914 scope.go:122] "RemoveContainer" containerID="107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986" Oct 30 21:30:10.913874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.913838 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:30:10.913983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.913968 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:30:10.914029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:10.914020 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:30:11.919579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:11.919545 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" event={"ID":"37547fb6-b031-4e65-ab9a-4bfa41ea3056","Type":"ContainerStarted","Data":"8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452"} Oct 30 21:30:12.098107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:12.098069 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:30:12.099416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:12.099389 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:30:12.099576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:12.099440 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:30:12.344813 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:30:12.344726 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:30:13.092907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:13.092308 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:30:13.092907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:13.092385 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:30:14.091907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:14.088248 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:30:14.091907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:14.088329 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:30:15.095913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:15.090950 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:30:15.095913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:15.091017 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:30:15.346314 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:15.346221 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:30:16.094914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.092946 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:30:16.094914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.093023 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:30:16.474002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.473960 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5"] Oct 30 21:30:16.501909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.498574 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.508558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.508526 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5"] Oct 30 21:30:16.550336 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.550305 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.550336 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.550337 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.550570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.550454 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.550570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.550487 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.550570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.550507 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.550718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.550561 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrgt\" (UniqueName: \"kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.554465 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.554445 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"openshift-service-ca.crt\"" Oct 30 21:30:16.554778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.554759 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"kube-root-ca.crt\"" Oct 30 21:30:16.554845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.554799 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"default-dockercfg-s2ptq\"" Oct 30 21:30:16.554913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.554850 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-07fa8e\"/\"simulator-w4-07fa8e-kserve-self-signed-certs\"" Oct 30 21:30:16.652226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652175 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652228 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652308 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652342 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652367 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652398 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrgt\" (UniqueName: \"kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652629 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.652752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.652723 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.653086 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.653063 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.654630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.654606 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.655240 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.655217 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.710589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.710561 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrgt\" (UniqueName: \"kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt\") pod \"simulator-w4-07fa8e-kserve-79df475566-rpqv5\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:16.830306 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:16.830220 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:17.088313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:17.088211 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:30:17.088313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:17.088280 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:30:17.259428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:17.259401 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5"] Oct 30 21:30:17.261118 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:30:17.261088 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc33b0585_3495_40c3_a1bd_50a7b5f3bff4.slice/crio-f6fdd69011ac15dd3323a1814bde89c00cdb601d60a8d86390dfca4129733ddd WatchSource:0}: Error finding container f6fdd69011ac15dd3323a1814bde89c00cdb601d60a8d86390dfca4129733ddd: Status 404 returned error can't find the container with id f6fdd69011ac15dd3323a1814bde89c00cdb601d60a8d86390dfca4129733ddd Oct 30 21:30:17.981274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:17.981238 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" event={"ID":"c33b0585-3495-40c3-a1bd-50a7b5f3bff4","Type":"ContainerStarted","Data":"d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3"} Oct 30 21:30:17.981768 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:17.981284 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" event={"ID":"c33b0585-3495-40c3-a1bd-50a7b5f3bff4","Type":"ContainerStarted","Data":"f6fdd69011ac15dd3323a1814bde89c00cdb601d60a8d86390dfca4129733ddd"} Oct 30 21:30:18.092293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:18.092261 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:30:18.093188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:18.093171 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 21:30:18.096937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:18.096867 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podStartSLOduration=2.096854643 podStartE2EDuration="2.096854643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:18.09602726 +0000 UTC m=+769.997891325" watchObservedRunningTime="2026-10-30 21:30:18.096854643 +0000 UTC m=+769.998718707" Oct 30 21:30:18.830707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:18.830671 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:26.831126 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:26.831088 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:26.843721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:26.843691 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:27.054729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:27.054699 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:30:30.898640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:30.898594 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:30:40.738965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.738934 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv"] Oct 30 21:30:40.742580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.742561 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.846814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.846771 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"kube-root-ca.crt\"" Oct 30 21:30:40.846814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.846808 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"openshift-service-ca.crt\"" Oct 30 21:30:40.847055 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.846817 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"default-dockercfg-qzw2w\"" Oct 30 21:30:40.847055 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.846921 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-021907\"/\"simulator-w5-021907-kserve-self-signed-certs\"" Oct 30 21:30:40.848720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.848700 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv"] Oct 30 21:30:40.858427 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.858402 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.858532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.858516 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.858589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.858575 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjbc\" (UniqueName: \"kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.858665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.858650 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.858707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.858684 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.858756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.858743 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960150 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960113 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960162 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960228 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960274 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjbc\" (UniqueName: \"kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960301 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960568 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960341 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960662 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960640 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960667 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.960747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.960710 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.962537 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.962515 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:40.962860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:40.962842 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:41.017152 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:41.017074 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjbc\" (UniqueName: \"kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc\") pod \"simulator-w5-021907-kserve-76d8b986c5-2dpkv\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:41.055225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:41.055189 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:41.442647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:41.442621 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv"] Oct 30 21:30:41.444665 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:30:41.444634 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod360d0a39_dfaf_4624_bc7a_b57d598e216b.slice/crio-d5bb7ccfe7a637a625f27360c4e731390bfbb0d8be4ce6634c979752d6e60795 WatchSource:0}: Error finding container d5bb7ccfe7a637a625f27360c4e731390bfbb0d8be4ce6634c979752d6e60795: Status 404 returned error can't find the container with id d5bb7ccfe7a637a625f27360c4e731390bfbb0d8be4ce6634c979752d6e60795 Oct 30 21:30:42.125670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:42.125630 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" event={"ID":"360d0a39-dfaf-4624-bc7a-b57d598e216b","Type":"ContainerStarted","Data":"743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3"} Oct 30 21:30:42.126114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:42.125679 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" event={"ID":"360d0a39-dfaf-4624-bc7a-b57d598e216b","Type":"ContainerStarted","Data":"d5bb7ccfe7a637a625f27360c4e731390bfbb0d8be4ce6634c979752d6e60795"} Oct 30 21:30:42.240518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:42.240465 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podStartSLOduration=2.240452465 podStartE2EDuration="2.240452465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:42.238030975 +0000 UTC m=+794.139895022" watchObservedRunningTime="2026-10-30 21:30:42.240452465 +0000 UTC m=+794.142316530" Oct 30 21:30:42.345720 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:30:42.345681 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:30:43.056227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:43.056197 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:49.099836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:49.099806 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:30:51.055438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:51.055380 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:51.068379 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:51.068355 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:30:51.184221 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:30:51.184192 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:31:06.200980 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:06.200945 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:31:12.252957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.252927 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n"] Oct 30 21:31:12.256963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.256943 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.331712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.331666 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.331922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.331773 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.331922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.331813 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.332059 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.331993 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.332156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.332139 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.332235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.332207 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsmz\" (UniqueName: \"kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.346064 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:31:12.346031 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:31:12.361617 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.361570 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"default-dockercfg-qk7t2\"" Oct 30 21:31:12.361617 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.361584 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"kube-root-ca.crt\"" Oct 30 21:31:12.361811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.361704 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"simulator-w0-1b5b5f-kserve-self-signed-certs\"" Oct 30 21:31:12.361868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.361836 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"openshift-service-ca.crt\"" Oct 30 21:31:12.363661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.363640 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n"] Oct 30 21:31:12.433204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433174 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433214 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsmz\" (UniqueName: \"kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433252 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433282 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433312 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433379 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433684 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433662 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.433870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433843 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.434054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.433957 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.436563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.436300 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.437307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.436915 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.490328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.490295 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsmz\" (UniqueName: \"kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz\") pod \"simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.569066 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.568987 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:12.945373 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:12.945351 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n"] Oct 30 21:31:12.947330 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:31:12.947303 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39953fdd_297a_4bc0_af16_9d4d9b68170a.slice/crio-ec7a9c1f1cda82089989a3b9dffd486d8bb8dae90935fb361323056e68269e8d WatchSource:0}: Error finding container ec7a9c1f1cda82089989a3b9dffd486d8bb8dae90935fb361323056e68269e8d: Status 404 returned error can't find the container with id ec7a9c1f1cda82089989a3b9dffd486d8bb8dae90935fb361323056e68269e8d Oct 30 21:31:13.313627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:13.313541 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" event={"ID":"39953fdd-297a-4bc0-af16-9d4d9b68170a","Type":"ContainerStarted","Data":"a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756"} Oct 30 21:31:13.313627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:13.313586 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" event={"ID":"39953fdd-297a-4bc0-af16-9d4d9b68170a","Type":"ContainerStarted","Data":"ec7a9c1f1cda82089989a3b9dffd486d8bb8dae90935fb361323056e68269e8d"} Oct 30 21:31:13.430242 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:13.429823 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podStartSLOduration=1.429803518 podStartE2EDuration="1.429803518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:13.428586822 +0000 UTC m=+825.330450887" watchObservedRunningTime="2026-10-30 21:31:13.429803518 +0000 UTC m=+825.331667586" Oct 30 21:31:13.569643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:13.569551 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:22.569867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:22.569833 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:22.582583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:22.582557 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:23.380761 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:23.380732 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:31:33.218181 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:33.218134 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:31:42.344558 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:31:42.344518 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:31:59.206580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.206547 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75"] Oct 30 21:31:59.210562 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.210545 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.311997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.311966 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctpg\" (UniqueName: \"kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.312171 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.312125 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.312213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.312198 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.312258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.312244 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.312316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.312301 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.312355 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.312346 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.313791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.313769 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"kube-root-ca.crt\"" Oct 30 21:31:59.313934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.313813 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"openshift-service-ca.crt\"" Oct 30 21:31:59.314036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.314016 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"default-dockercfg-hqthv\"" Oct 30 21:31:59.314118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.314101 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-446ca2\"/\"premium-w2-446ca2-kserve-self-signed-certs\"" Oct 30 21:31:59.315809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.315792 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75"] Oct 30 21:31:59.413291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413255 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctpg\" (UniqueName: \"kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.413505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413384 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.413505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413417 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.413505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413470 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.413663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413505 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.413663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413552 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.413922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413899 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.414004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413899 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.414004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.413981 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.415977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.415950 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.416157 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.416142 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.475922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.475819 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctpg\" (UniqueName: \"kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg\") pod \"premium-w2-446ca2-kserve-5499b9fd4-vdn75\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.523022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.522987 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:31:59.920040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:31:59.920011 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75"] Oct 30 21:31:59.922179 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:31:59.922153 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a117ec3_78ec_418e_aec2_f333f54d2935.slice/crio-c2198d50b306945f94c06f9a7180911952d81c7bb3c4e4da7f6455a9c2e7809f WatchSource:0}: Error finding container c2198d50b306945f94c06f9a7180911952d81c7bb3c4e4da7f6455a9c2e7809f: Status 404 returned error can't find the container with id c2198d50b306945f94c06f9a7180911952d81c7bb3c4e4da7f6455a9c2e7809f Oct 30 21:32:00.640422 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:00.640370 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" event={"ID":"8a117ec3-78ec-418e-aec2-f333f54d2935","Type":"ContainerStarted","Data":"b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db"} Oct 30 21:32:00.640422 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:00.640426 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" event={"ID":"8a117ec3-78ec-418e-aec2-f333f54d2935","Type":"ContainerStarted","Data":"c2198d50b306945f94c06f9a7180911952d81c7bb3c4e4da7f6455a9c2e7809f"} Oct 30 21:32:00.754572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:00.754519 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podStartSLOduration=1.7545056030000001 podStartE2EDuration="1.754505603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:00.753919601 +0000 UTC m=+872.655783663" watchObservedRunningTime="2026-10-30 21:32:00.754505603 +0000 UTC m=+872.656369699" Oct 30 21:32:01.523627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:01.523589 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:32:02.562177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:02.562144 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:32:09.523233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:09.523192 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:32:09.536418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:09.536388 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:32:09.702072 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:09.702041 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:32:12.345992 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:32:12.345949 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:32:23.048347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:23.048308 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:32:28.333535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:28.333505 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:32:28.336681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:28.336662 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:32:42.345796 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:32:42.345712 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:32:52.276790 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:32:52.276746 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:33:12.345481 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:33:12.345437 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:33:19.224233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:33:19.224200 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:33:42.345833 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:33:42.345795 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:34:02.449082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:02.449051 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:34:12.345083 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:34:12.345004 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:34:33.350440 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:33.350407 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59"] Oct 30 21:34:33.350990 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:33.350592 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" containerID="cri-o://f7a16265cb8eccb62fd3d23c587efdad4d21bf00c396a5e424ccd1b02abe96f2" gracePeriod=30 Oct 30 21:34:38.555965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.555934 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb"] Oct 30 21:34:38.560288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.560263 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.575076 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.575047 5914 generic.go:356] "Generic (PLEG): container finished" podID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerID="f7a16265cb8eccb62fd3d23c587efdad4d21bf00c396a5e424ccd1b02abe96f2" exitCode=0 Oct 30 21:34:38.575221 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.575118 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" event={"ID":"178f0036-19d7-4e38-b37e-e60ad4fec147","Type":"ContainerDied","Data":"f7a16265cb8eccb62fd3d23c587efdad4d21bf00c396a5e424ccd1b02abe96f2"} Oct 30 21:34:38.610219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.610186 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:34:38.663293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.663257 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb"] Oct 30 21:34:38.664249 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.664232 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"distinct-w0-1b5b5f-kserve-self-signed-certs\"" Oct 30 21:34:38.667798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.667781 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.667849 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.667839 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.667952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.667930 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.668001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.667989 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fncg\" (UniqueName: \"kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.668053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.668040 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.668088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.668064 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.769048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769015 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769098 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769118 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769139 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769210 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769248 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769281 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769336 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.769416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769372 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769411 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy\") pod \"178f0036-19d7-4e38-b37e-e60ad4fec147\" (UID: \"178f0036-19d7-4e38-b37e-e60ad4fec147\") " Oct 30 21:34:38.769674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769474 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.769674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769539 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.769790 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769721 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.769790 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769738 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.769790 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769768 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:38.769983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769923 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.769983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.769964 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.770254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770125 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fncg\" (UniqueName: \"kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.770254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770157 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.770254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770187 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.770460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770362 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.770460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770454 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178f0036-19d7-4e38-b37e-e60ad4fec147-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.770460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770455 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.770720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770469 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.770720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770485 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.770720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770500 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.770913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.770790 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.771139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.771117 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.771992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.771957 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:34:38.772112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.772094 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "kube-api-access-lzgfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:38.772254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.772240 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:38.772552 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.772526 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy" pod "178f0036-19d7-4e38-b37e-e60ad4fec147" (UID: "178f0036-19d7-4e38-b37e-e60ad4fec147"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.772642 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.772598 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.773567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.773548 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.830957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.830924 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fncg\" (UniqueName: \"kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg\") pod \"distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:38.871703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.871670 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgfc\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-kube-api-access-lzgfc\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.871703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.871698 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.871703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.871709 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.871946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.871718 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178f0036-19d7-4e38-b37e-e60ad4fec147-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.874383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:38.874366 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:39.248116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.248088 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb"] Oct 30 21:34:39.251004 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:34:39.250976 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod570faa65_6036_4f69_b260_832f77f691d4.slice/crio-f2f98e93a920ff93aab1ffe7f7e60450bdd9faff3f4ba3556d11a928b6d000ab WatchSource:0}: Error finding container f2f98e93a920ff93aab1ffe7f7e60450bdd9faff3f4ba3556d11a928b6d000ab: Status 404 returned error can't find the container with id f2f98e93a920ff93aab1ffe7f7e60450bdd9faff3f4ba3556d11a928b6d000ab Oct 30 21:34:39.588825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.588740 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" Oct 30 21:34:39.588825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.588767 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59" event={"ID":"178f0036-19d7-4e38-b37e-e60ad4fec147","Type":"ContainerDied","Data":"c6bf310cff4b9c99f6e3d7d9980e2e8bd1c88c06347337c1a48429d78e1542c0"} Oct 30 21:34:39.589366 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.588865 5914 scope.go:122] "RemoveContainer" containerID="f7a16265cb8eccb62fd3d23c587efdad4d21bf00c396a5e424ccd1b02abe96f2" Oct 30 21:34:39.590650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.590612 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" event={"ID":"570faa65-6036-4f69-b260-832f77f691d4","Type":"ContainerStarted","Data":"f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7"} Oct 30 21:34:39.590756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.590652 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" event={"ID":"570faa65-6036-4f69-b260-832f77f691d4","Type":"ContainerStarted","Data":"f2f98e93a920ff93aab1ffe7f7e60450bdd9faff3f4ba3556d11a928b6d000ab"} Oct 30 21:34:39.763393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.763342 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podStartSLOduration=1.7633277889999999 podStartE2EDuration="1.763327789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:39.762418109 +0000 UTC m=+1031.664282174" watchObservedRunningTime="2026-10-30 21:34:39.763327789 +0000 UTC m=+1031.665191854" Oct 30 21:34:39.876478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.876417 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59"] Oct 30 21:34:39.876745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.876727 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:39.890737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:39.890683 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-rnc59"] Oct 30 21:34:40.216688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:40.216612 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" path="/var/lib/kubelet/pods/178f0036-19d7-4e38-b37e-e60ad4fec147/volumes" Oct 30 21:34:42.345731 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:34:42.345696 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:34:45.863532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.863494 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd"] Oct 30 21:34:45.864057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.864043 5914 cpu_manager.go:414] "containerMap: removing container" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" Oct 30 21:34:45.864107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.864058 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" Oct 30 21:34:45.864154 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.864145 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="178f0036-19d7-4e38-b37e-e60ad4fec147" containerName="istio-proxy" Oct 30 21:34:45.867265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.867246 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.919780 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.919740 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-89c0fd-openshift-default-dockercfg-kzh9g\"" Oct 30 21:34:45.946339 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946301 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8xkz\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946530 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946355 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946530 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946456 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946536 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946633 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946727 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946659 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946745 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946803 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946790 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.946849 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.946811 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:45.971677 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:45.971646 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd"] Oct 30 21:34:46.047973 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.047933 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048168 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048019 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048168 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048046 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048168 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048081 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8xkz\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048214 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048281 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048320 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048380 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048429 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048435 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048454 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048657 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048754 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.048994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.048976 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.050647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.050629 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.051124 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.051102 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.105018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.104980 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.105145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.105042 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8xkz\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.180340 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.180304 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:46.180544 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.180308 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:34:46.618016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.617989 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd"] Oct 30 21:34:46.620020 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:34:46.619997 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bc112e_9447_45bb_a7de_13694d0b4c2d.slice/crio-6f2e2b9cd0e5fe62776350d1d75e1c41b23237edeb8984d94a66cb61c7863c00 WatchSource:0}: Error finding container 6f2e2b9cd0e5fe62776350d1d75e1c41b23237edeb8984d94a66cb61c7863c00: Status 404 returned error can't find the container with id 6f2e2b9cd0e5fe62776350d1d75e1c41b23237edeb8984d94a66cb61c7863c00 Oct 30 21:34:46.622485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.622458 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:34:46.622542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.622526 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:34:46.622582 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.622555 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:34:46.637199 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:46.637173 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" event={"ID":"c0bc112e-9447-45bb-a7de-13694d0b4c2d","Type":"ContainerStarted","Data":"6f2e2b9cd0e5fe62776350d1d75e1c41b23237edeb8984d94a66cb61c7863c00"} Oct 30 21:34:47.647478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:47.647437 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" event={"ID":"c0bc112e-9447-45bb-a7de-13694d0b4c2d","Type":"ContainerStarted","Data":"6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b"} Oct 30 21:34:47.813890 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:47.813825 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podStartSLOduration=2.81380637 podStartE2EDuration="2.81380637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:47.811457521 +0000 UTC m=+1039.713321624" watchObservedRunningTime="2026-10-30 21:34:47.81380637 +0000 UTC m=+1039.715670436" Oct 30 21:34:48.195918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:48.186592 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:48.195918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:48.186739 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:34:48.195918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:48.186780 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:34:48.893908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:48.887212 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:48.907925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:48.907864 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:49.187968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:49.185271 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:34:49.187968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:49.185329 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:34:49.670635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:49.670605 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:34:50.192907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:50.192224 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:34:50.192907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:50.192301 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:34:51.188915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:51.185218 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:34:51.188915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:51.185292 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:34:52.187474 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:52.187435 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:34:52.188115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:52.188078 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:34:53.366524 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:53.366424 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:53.367226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:53.367174 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:34:53.368209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:34:53.368185 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:35:12.345840 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:35:12.345796 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:35:16.305981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.304329 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k"] Oct 30 21:35:16.310187 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.310161 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.364697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.364655 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k"] Oct 30 21:35:16.370920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.370861 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.371131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.370973 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.371131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.371045 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.371131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.371082 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcls\" (UniqueName: \"kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.371311 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.371152 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.371311 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.371178 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.411602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.411574 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"kube-root-ca.crt\"" Oct 30 21:35:16.411602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.411590 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"default-dockercfg-45994\"" Oct 30 21:35:16.411602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.411590 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"openshift-service-ca.crt\"" Oct 30 21:35:16.413432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.413418 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e4dadd\"/\"distinct-w1-e4dadd-kserve-self-signed-certs\"" Oct 30 21:35:16.475931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.475766 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.476338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.476309 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.476477 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.476373 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.476477 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.476451 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.476591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.476488 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcls\" (UniqueName: \"kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.476591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.476561 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.476591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.476585 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.477839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.477813 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.478038 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.477837 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.481513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.481195 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.481931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.481857 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.533995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.533965 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcls\" (UniqueName: \"kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls\") pod \"distinct-w1-e4dadd-kserve-669f669478-dhw5k\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:16.625706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.625675 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:17.000019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:16.999990 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k"] Oct 30 21:35:17.001551 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:35:17.001522 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8868cef_5ba6_485c_b2b5_f66b52d6bd0d.slice/crio-e3f302f80936389fa984c745f40008aad6e7c37dbb6e550de3102e9d23182e5b WatchSource:0}: Error finding container e3f302f80936389fa984c745f40008aad6e7c37dbb6e550de3102e9d23182e5b: Status 404 returned error can't find the container with id e3f302f80936389fa984c745f40008aad6e7c37dbb6e550de3102e9d23182e5b Oct 30 21:35:17.367712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:17.367625 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:35:17.854651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:17.854618 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" event={"ID":"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d","Type":"ContainerStarted","Data":"15e8bfe71f867c7d6f0511c7b6d5a6602610a4545afd7ba203a13ef11e71c898"} Oct 30 21:35:17.854831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:17.854660 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" event={"ID":"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d","Type":"ContainerStarted","Data":"e3f302f80936389fa984c745f40008aad6e7c37dbb6e550de3102e9d23182e5b"} Oct 30 21:35:17.969922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:17.969857 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podStartSLOduration=1.969841729 podStartE2EDuration="1.969841729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:17.968142666 +0000 UTC m=+1069.870006730" watchObservedRunningTime="2026-10-30 21:35:17.969841729 +0000 UTC m=+1069.871705794" Oct 30 21:35:18.625797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:18.625766 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:26.626208 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:26.626174 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:26.639444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:26.639419 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:26.920195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:26.920120 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:35:42.346327 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:35:42.346243 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:35:45.187814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.187779 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx"] Oct 30 21:35:45.191585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.191569 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.295171 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.295142 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"kube-root-ca.crt\"" Oct 30 21:35:45.295348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.295197 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"distinct-w6-89c0fd-kserve-self-signed-certs\"" Oct 30 21:35:45.295348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.295216 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"default-dockercfg-mmr5s\"" Oct 30 21:35:45.295348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.295265 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"openshift-service-ca.crt\"" Oct 30 21:35:45.297382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.297358 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx"] Oct 30 21:35:45.355910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.351560 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.356295 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.356259 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47v6t\" (UniqueName: \"kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.356491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.356471 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.356680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.356665 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.356841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.356827 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.356977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.356962 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458223 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v6t\" (UniqueName: \"kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458266 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458318 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458361 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458382 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458460 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458746 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458770 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.458986 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.458830 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.461007 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.460985 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.461140 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.461123 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.516085 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.516054 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v6t\" (UniqueName: \"kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t\") pod \"distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:45.803502 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:45.803397 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:46.178008 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:46.177982 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx"] Oct 30 21:35:46.180566 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:35:46.180534 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c4cee3d_3fd2_453a_b01b_0499c6d2a773.slice/crio-1e26845c0d3da45dcf726f0ac56bf80a7afc6e16f732d1905348b087bfe063ba WatchSource:0}: Error finding container 1e26845c0d3da45dcf726f0ac56bf80a7afc6e16f732d1905348b087bfe063ba: Status 404 returned error can't find the container with id 1e26845c0d3da45dcf726f0ac56bf80a7afc6e16f732d1905348b087bfe063ba Oct 30 21:35:47.038319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:47.038282 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" event={"ID":"9c4cee3d-3fd2-453a-b01b-0499c6d2a773","Type":"ContainerStarted","Data":"649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b"} Oct 30 21:35:47.038319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:47.038323 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" event={"ID":"9c4cee3d-3fd2-453a-b01b-0499c6d2a773","Type":"ContainerStarted","Data":"1e26845c0d3da45dcf726f0ac56bf80a7afc6e16f732d1905348b087bfe063ba"} Oct 30 21:35:47.156128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:47.156074 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podStartSLOduration=2.156058624 podStartE2EDuration="2.156058624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:47.153974075 +0000 UTC m=+1099.055838141" watchObservedRunningTime="2026-10-30 21:35:47.156058624 +0000 UTC m=+1099.057922700" Oct 30 21:35:47.804245 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:47.804213 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:54.693817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:54.693782 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:35:55.804733 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:55.804690 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:55.819363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:55.819336 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:35:56.120036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:35:56.119949 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:36:04.048147 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:04.048116 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j"] Oct 30 21:36:04.048553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:04.048319 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" containerID="cri-o://d523295b8c19fa78ba74ec27235ac8245266b5d5ee04f47bf5df4b6540a0c58b" gracePeriod=30 Oct 30 21:36:09.222712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.222677 5914 generic.go:356] "Generic (PLEG): container finished" podID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerID="d523295b8c19fa78ba74ec27235ac8245266b5d5ee04f47bf5df4b6540a0c58b" exitCode=0 Oct 30 21:36:09.223418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.222728 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" event={"ID":"aa3a3936-5efb-4bfd-a45f-1f62cf689329","Type":"ContainerDied","Data":"d523295b8c19fa78ba74ec27235ac8245266b5d5ee04f47bf5df4b6540a0c58b"} Oct 30 21:36:09.335449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.335424 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:36:09.393779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.393751 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.393987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.393804 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.393987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.393848 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.393987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.393935 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.393987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.393965 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.394203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394002 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.394203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394078 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:09.394203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394092 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.394203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394173 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.394412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394242 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket\") pod \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\" (UID: \"aa3a3936-5efb-4bfd-a45f-1f62cf689329\") " Oct 30 21:36:09.394412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394167 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:09.394412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394275 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:09.394412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394385 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:09.394670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394648 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:09.394952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394932 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.395019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394960 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.395019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394976 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.395019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.394991 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.395019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.395006 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.396372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.396341 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:09.396479 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.396449 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:36:09.396577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.396561 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "kube-api-access-q9dcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:09.396638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.396624 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy" pod "aa3a3936-5efb-4bfd-a45f-1f62cf689329" (UID: "aa3a3936-5efb-4bfd-a45f-1f62cf689329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:09.496464 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.496417 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.496628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.496473 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.496628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.496485 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9dcx\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-kube-api-access-q9dcx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:09.496628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:09.496497 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa3a3936-5efb-4bfd-a45f-1f62cf689329-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:36:10.230814 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:10.230792 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" Oct 30 21:36:10.231157 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:10.230785 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j" event={"ID":"aa3a3936-5efb-4bfd-a45f-1f62cf689329","Type":"ContainerDied","Data":"6a53f65013352e6c0075908b36b669881cd86ea33f23f79d7cae66e5c69a4cd0"} Oct 30 21:36:10.231157 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:10.230942 5914 scope.go:122] "RemoveContainer" containerID="d523295b8c19fa78ba74ec27235ac8245266b5d5ee04f47bf5df4b6540a0c58b" Oct 30 21:36:10.396114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:10.396079 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j"] Oct 30 21:36:10.445507 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:10.445467 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-2462j"] Oct 30 21:36:12.217127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:12.217091 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" path="/var/lib/kubelet/pods/aa3a3936-5efb-4bfd-a45f-1f62cf689329/volumes" Oct 30 21:36:12.345066 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:36:12.345034 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:36:19.159335 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.159298 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj"] Oct 30 21:36:19.160000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.159848 5914 cpu_manager.go:414] "containerMap: removing container" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" Oct 30 21:36:19.160000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.159861 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" Oct 30 21:36:19.160127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.160055 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3a3936-5efb-4bfd-a45f-1f62cf689329" containerName="istio-proxy" Oct 30 21:36:19.163624 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.163603 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.215528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.215501 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-021907-openshift-default-dockercfg-dm7nx\"" Oct 30 21:36:19.267742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.267710 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj"] Oct 30 21:36:19.315105 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315072 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315153 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjrn\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315274 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315338 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315403 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315489 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315529 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315569 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.315651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.315598 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.416874 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417044 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjrn\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417132 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417191 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417230 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417315 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417360 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417388 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417560 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417417 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417637 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417603 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417699 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417669 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417758 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417722 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.417855 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.417835 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.418397 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.418372 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.419621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.419597 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.419741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.419723 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.473935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.473867 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.474107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.474083 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjrn\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.477705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.477677 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:19.923964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.923940 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj"] Oct 30 21:36:19.925658 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:36:19.925631 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb985fa53_55d1_40e6_a488_8a30ef36e1d2.slice/crio-36602301ab9015afc35f9cd90800051bb44afb7ff4f3f404a3e62c6348feec37 WatchSource:0}: Error finding container 36602301ab9015afc35f9cd90800051bb44afb7ff4f3f404a3e62c6348feec37: Status 404 returned error can't find the container with id 36602301ab9015afc35f9cd90800051bb44afb7ff4f3f404a3e62c6348feec37 Oct 30 21:36:19.927949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.927917 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:36:19.928037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.927990 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:36:19.928037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:19.928020 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:36:20.299588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:20.299481 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" event={"ID":"b985fa53-55d1-40e6-a488-8a30ef36e1d2","Type":"ContainerStarted","Data":"6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61"} Oct 30 21:36:20.299588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:20.299573 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" event={"ID":"b985fa53-55d1-40e6-a488-8a30ef36e1d2","Type":"ContainerStarted","Data":"36602301ab9015afc35f9cd90800051bb44afb7ff4f3f404a3e62c6348feec37"} Oct 30 21:36:20.416994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:20.416934 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podStartSLOduration=1.416918287 podStartE2EDuration="1.416918287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:20.415017406 +0000 UTC m=+1132.316881472" watchObservedRunningTime="2026-10-30 21:36:20.416918287 +0000 UTC m=+1132.318782351" Oct 30 21:36:20.490422 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:20.490366 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:20.499162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:20.499078 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:20.499162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:20.499141 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:21.496907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:21.495252 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:21.496907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:21.495317 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:22.493907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:22.489265 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:22.493907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:22.489326 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:23.490268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:23.490233 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:23.490943 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:23.490912 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:24.491917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:24.488948 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:24.491917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:24.489004 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:25.488366 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:25.488323 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:25.488681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:25.488651 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:26.478065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:26.478031 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:36:26.478524 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:26.478087 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:36:27.482428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:27.482399 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:27.482812 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:27.482755 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:27.483449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:27.483431 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:36:30.265401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.265366 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz"] Oct 30 21:36:30.270711 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.270691 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.326623 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.326569 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz"] Oct 30 21:36:30.373335 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.373303 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1b5b5f\"/\"distinct-2-w0-1b5b5f-kserve-self-signed-certs\"" Oct 30 21:36:30.466564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.466531 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.466741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.466592 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.466741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.466704 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.466741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.466733 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.466843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.466751 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrr9\" (UniqueName: \"kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.466914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.466876 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.567928 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.567998 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568041 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568059 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568161 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrr9\" (UniqueName: \"kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568257 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568371 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568437 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.568514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.568493 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.570392 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.570366 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.570710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.570692 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.636578 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.636541 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrr9\" (UniqueName: \"kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9\") pod \"distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:30.883532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:30.883499 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:31.254959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:31.254863 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz"] Oct 30 21:36:31.256556 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:36:31.256533 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad107193_e9ed_4053_8855_d8aa7e53d9f6.slice/crio-08c377a979a12662a1a3fb5893c367a7cb54b767f15699b3c9567a7a9194b4e8 WatchSource:0}: Error finding container 08c377a979a12662a1a3fb5893c367a7cb54b767f15699b3c9567a7a9194b4e8: Status 404 returned error can't find the container with id 08c377a979a12662a1a3fb5893c367a7cb54b767f15699b3c9567a7a9194b4e8 Oct 30 21:36:31.362659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:31.362623 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" event={"ID":"ad107193-e9ed-4053-8855-d8aa7e53d9f6","Type":"ContainerStarted","Data":"a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef"} Oct 30 21:36:31.362659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:31.362666 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" event={"ID":"ad107193-e9ed-4053-8855-d8aa7e53d9f6","Type":"ContainerStarted","Data":"08c377a979a12662a1a3fb5893c367a7cb54b767f15699b3c9567a7a9194b4e8"} Oct 30 21:36:31.478197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:31.478144 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podStartSLOduration=1.478130027 podStartE2EDuration="1.478130027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:31.477304741 +0000 UTC m=+1143.379168811" watchObservedRunningTime="2026-10-30 21:36:31.478130027 +0000 UTC m=+1143.379994092" Oct 30 21:36:31.883746 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:31.883706 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:32.842409 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:32.842374 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:36:40.884540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:40.884504 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:40.897157 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:40.897132 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:41.430501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:36:41.430472 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:36:42.346159 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:36:42.346123 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:37:01.889849 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:01.889816 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:37:12.345578 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:37:12.345496 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:37:28.376449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:28.376420 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:37:28.378285 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:28.378263 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:37:31.197763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:31.197731 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:37:38.149112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.149077 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb"] Oct 30 21:37:38.153282 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.153264 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.205312 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.205282 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-07fa8e-openshift-default-dockercfg-whqhj\"" Oct 30 21:37:38.235131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235093 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235137 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235171 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235310 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235387 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235511 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235463 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwbs\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235565 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235591 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.235630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.235607 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.257304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.257271 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb"] Oct 30 21:37:38.336954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.336922 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337000 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337041 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwbs\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337127 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337144 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337161 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337205 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337241 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337280 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337364 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337531 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337608 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337654 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.337933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.337909 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.339645 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.339623 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.339829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.339812 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.393825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.393797 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.394046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.394026 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwbs\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.467496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.467418 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:38.894481 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.894450 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb"] Oct 30 21:37:38.896458 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:37:38.896433 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804d17fd_e5f6_4ffa_b3b2_7d044356142b.slice/crio-4383f4396e71219dd3cc50cc096ae241f605f2b145efe48b139e4d6ed817efdc WatchSource:0}: Error finding container 4383f4396e71219dd3cc50cc096ae241f605f2b145efe48b139e4d6ed817efdc: Status 404 returned error can't find the container with id 4383f4396e71219dd3cc50cc096ae241f605f2b145efe48b139e4d6ed817efdc Oct 30 21:37:38.898990 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.898955 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:37:38.899074 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.899026 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:37:38.899074 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:38.899060 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:37:39.764197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:39.764157 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" event={"ID":"804d17fd-e5f6-4ffa-b3b2-7d044356142b","Type":"ContainerStarted","Data":"429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8"} Oct 30 21:37:39.764598 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:39.764222 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" event={"ID":"804d17fd-e5f6-4ffa-b3b2-7d044356142b","Type":"ContainerStarted","Data":"4383f4396e71219dd3cc50cc096ae241f605f2b145efe48b139e4d6ed817efdc"} Oct 30 21:37:39.899906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:39.899826 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podStartSLOduration=1.8998061389999998 podStartE2EDuration="1.899806139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:39.89940943 +0000 UTC m=+1211.801273496" watchObservedRunningTime="2026-10-30 21:37:39.899806139 +0000 UTC m=+1211.801670205" Oct 30 21:37:40.467795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:40.467751 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:40.469494 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:40.469468 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:37:40.469616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:40.469522 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:37:41.468173 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:41.468136 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:37:41.468548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:41.468200 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:37:42.344332 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:37:42.344296 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:37:42.468235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:42.468200 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:37:42.468689 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:42.468261 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:37:43.468759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:43.468727 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:37:43.469183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:43.468781 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:37:44.468607 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:44.468569 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:37:44.468916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:44.468853 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:37:45.468037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:45.467995 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:37:45.468229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:45.468068 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:37:46.473029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:46.472999 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:46.473487 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:46.473377 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:46.474059 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:46.474043 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:37:57.535424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.535388 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb"] Oct 30 21:37:57.539600 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.539581 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.606683 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.606649 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvndn\" (UniqueName: \"kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.606852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.606709 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.606852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.606820 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.607020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.606900 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.607020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.606962 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.607127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.607094 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.643200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.643175 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-89c0fd\"/\"distinct-2-w6-89c0fd-kserve-self-signed-certs\"" Oct 30 21:37:57.645532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.645509 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb"] Oct 30 21:37:57.708802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.708766 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.709567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.709539 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.709759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.709744 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.709864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.709849 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.714851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.713819 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.714851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.714074 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.714851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.714268 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvndn\" (UniqueName: \"kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.715125 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.715039 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.715607 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.715583 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.737123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.736370 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.737123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.737070 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.773669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.773641 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvndn\" (UniqueName: \"kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn\") pod \"distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:57.853098 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:57.853018 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:37:58.226933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:58.226905 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb"] Oct 30 21:37:58.229054 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:37:58.229031 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d2491f_fbab_4843_9793_17fbd81dcb8c.slice/crio-9680047fdde2e0b7130c9e28b8f2f1af18fedce40af3cb16f5e0f9fda45079b3 WatchSource:0}: Error finding container 9680047fdde2e0b7130c9e28b8f2f1af18fedce40af3cb16f5e0f9fda45079b3: Status 404 returned error can't find the container with id 9680047fdde2e0b7130c9e28b8f2f1af18fedce40af3cb16f5e0f9fda45079b3 Oct 30 21:37:58.980669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:58.980631 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" event={"ID":"a0d2491f-fbab-4843-9793-17fbd81dcb8c","Type":"ContainerStarted","Data":"59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef"} Oct 30 21:37:58.981159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:58.980677 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" event={"ID":"a0d2491f-fbab-4843-9793-17fbd81dcb8c","Type":"ContainerStarted","Data":"9680047fdde2e0b7130c9e28b8f2f1af18fedce40af3cb16f5e0f9fda45079b3"} Oct 30 21:37:59.095685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:59.095632 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podStartSLOduration=2.095617812 podStartE2EDuration="2.095617812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:59.094771155 +0000 UTC m=+1230.996635203" watchObservedRunningTime="2026-10-30 21:37:59.095617812 +0000 UTC m=+1230.997481877" Oct 30 21:37:59.854143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:37:59.854109 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:38:07.853346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:07.853311 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:38:07.866487 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:07.866460 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:38:08.044552 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.044523 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:38:08.080650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.080618 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6"] Oct 30 21:38:08.084845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.084828 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.140304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.140263 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-446ca2-openshift-default-dockercfg-2jv4g\"" Oct 30 21:38:08.143282 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.143255 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6"] Oct 30 21:38:08.151021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.150996 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151170 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151041 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151156 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151197 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151284 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151417 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151400 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151425 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151455 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.151585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.151506 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsv4p\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.252968 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253001 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253035 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253080 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsv4p\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253145 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253170 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253236 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253258 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253308 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253433 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253629 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253693 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.253867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253847 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.254014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.253994 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.255565 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.255544 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.255786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.255767 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.310689 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.310645 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.310839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.310769 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsv4p\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.399207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.399120 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:08.829401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.829376 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6"] Oct 30 21:38:08.831023 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:38:08.830986 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dd13dc_19c6_4439_94f6_128a040d28ab.slice/crio-e4f08ffb5edd8bd18ed6ef8a2e5cee76241c44b3d12581645a10c22c11d205e0 WatchSource:0}: Error finding container e4f08ffb5edd8bd18ed6ef8a2e5cee76241c44b3d12581645a10c22c11d205e0: Status 404 returned error can't find the container with id e4f08ffb5edd8bd18ed6ef8a2e5cee76241c44b3d12581645a10c22c11d205e0 Oct 30 21:38:08.833407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.833369 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:08.833534 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.833444 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:08.833534 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:08.833503 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:09.041182 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:09.041146 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" event={"ID":"79dd13dc-19c6-4439-94f6-128a040d28ab","Type":"ContainerStarted","Data":"0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047"} Oct 30 21:38:09.041580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:09.041203 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" event={"ID":"79dd13dc-19c6-4439-94f6-128a040d28ab","Type":"ContainerStarted","Data":"e4f08ffb5edd8bd18ed6ef8a2e5cee76241c44b3d12581645a10c22c11d205e0"} Oct 30 21:38:09.160961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:09.160909 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podStartSLOduration=1.160893468 podStartE2EDuration="1.160893468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:09.159177714 +0000 UTC m=+1241.061041782" watchObservedRunningTime="2026-10-30 21:38:09.160893468 +0000 UTC m=+1241.062757579" Oct 30 21:38:09.399643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:09.399594 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:09.401602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:09.401571 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:09.401767 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:09.401629 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:10.400415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:10.400362 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:10.400955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:10.400434 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:11.400138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:11.400096 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:11.400360 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:11.400161 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:12.346712 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:38:12.346672 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:38:12.399875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:12.399825 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:12.400088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:12.399917 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:13.400482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:13.400440 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:13.400943 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:13.400497 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:14.399641 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:14.399605 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:14.399860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:14.399667 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:15.399635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:15.399601 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 21:38:15.400071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:15.399657 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 21:38:16.404444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:16.404414 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:16.404858 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:16.404793 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:16.405559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:16.405537 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:38:17.568675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:17.568646 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:38:38.613967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.613864 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x"] Oct 30 21:38:38.617903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.617871 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.706940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.706871 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.706998 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzlg\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707098 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707215 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707274 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707369 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707408 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707450 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.707531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.707485 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808355 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808318 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808402 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808448 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808499 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808520 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808554 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808593 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808741 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.808872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808783 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzlg\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.809077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.808965 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.809077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.809050 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.809176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.809098 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.809254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.809232 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.809650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.809632 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.810996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.810979 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.811214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.811192 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.822284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.822255 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x"] Oct 30 21:38:38.866130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.866052 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.866278 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.866142 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzlg\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:38.930963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:38.930931 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:39.383660 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:39.383248 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x"] Oct 30 21:38:39.411909 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:38:39.409234 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8681933e_71c3_44b2_ba83_71d851812d50.slice/crio-3b40f0dab9991b71129701d6a67bf3e1100759efc0c716b854f35ed0d2f79fc6 WatchSource:0}: Error finding container 3b40f0dab9991b71129701d6a67bf3e1100759efc0c716b854f35ed0d2f79fc6: Status 404 returned error can't find the container with id 3b40f0dab9991b71129701d6a67bf3e1100759efc0c716b854f35ed0d2f79fc6 Oct 30 21:38:39.426207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:39.424492 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:39.426207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:39.424584 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:39.426207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:39.424626 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:40.329404 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:40.329361 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" event={"ID":"8681933e-71c3-44b2-ba83-71d851812d50","Type":"ContainerStarted","Data":"e1084a421bde95c150fb3e2b2b22a87cbf677144b712b2f0f445bf6f9de536ca"} Oct 30 21:38:40.329983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:40.329431 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" event={"ID":"8681933e-71c3-44b2-ba83-71d851812d50","Type":"ContainerStarted","Data":"3b40f0dab9991b71129701d6a67bf3e1100759efc0c716b854f35ed0d2f79fc6"} Oct 30 21:38:40.459905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:40.459249 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podStartSLOduration=2.459228497 podStartE2EDuration="2.459228497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:40.455326777 +0000 UTC m=+1272.357190843" watchObservedRunningTime="2026-10-30 21:38:40.459228497 +0000 UTC m=+1272.361092566" Oct 30 21:38:40.931115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:40.931078 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:40.932855 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:40.932825 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:38:40.933013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:40.932913 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:38:41.931579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:41.931545 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:38:41.932031 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:41.931596 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:38:42.347105 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:38:42.347010 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:38:42.931896 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:42.931854 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:38:42.932277 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:42.931942 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:38:43.931352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:43.931315 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:38:43.931538 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:43.931372 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:38:44.932072 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:44.932036 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:38:44.932466 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:44.932091 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:38:45.931517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:45.931476 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:38:45.931707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:45.931545 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:38:46.965370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:46.965335 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:46.966258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:46.966235 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:46.967365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:46.967341 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:38:50.311173 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.311143 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq"] Oct 30 21:38:50.315312 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.315283 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407090 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407127 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407222 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407278 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407416 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmmz\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407566 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407543 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407615 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407593 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407675 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.407729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.407709 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.430793 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.430768 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq"] Oct 30 21:38:50.509048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509014 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509051 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509091 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509116 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509154 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509207 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509349 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmmz\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509466 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509479 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509505 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509698 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509569 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509733 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.509822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.509791 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.510183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.510123 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.511686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.511644 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.511894 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.511859 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.566214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.566143 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.566437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.566413 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmmz\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:50.629119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:50.629087 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:51.070261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.070236 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq"] Oct 30 21:38:51.072030 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:38:51.072004 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39efa30_1507_4ebb_a488_1dd88d1a22a6.slice/crio-852eb1c1da1bb7e218a804c237ad77a367910fd3d8f4da75ffaa8a855f85c08f WatchSource:0}: Error finding container 852eb1c1da1bb7e218a804c237ad77a367910fd3d8f4da75ffaa8a855f85c08f: Status 404 returned error can't find the container with id 852eb1c1da1bb7e218a804c237ad77a367910fd3d8f4da75ffaa8a855f85c08f Oct 30 21:38:51.074535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.074502 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:51.074633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.074571 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:51.074633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.074603 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:38:51.400563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.400528 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" event={"ID":"e39efa30-1507-4ebb-a488-1dd88d1a22a6","Type":"ContainerStarted","Data":"425783994b5b50b7df7db3754400315f3a613a90573c04bb387b5f21242f51e3"} Oct 30 21:38:51.400987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.400583 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" event={"ID":"e39efa30-1507-4ebb-a488-1dd88d1a22a6","Type":"ContainerStarted","Data":"852eb1c1da1bb7e218a804c237ad77a367910fd3d8f4da75ffaa8a855f85c08f"} Oct 30 21:38:51.521316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.521263 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podStartSLOduration=1.521247887 podStartE2EDuration="1.521247887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:51.517426114 +0000 UTC m=+1283.419290180" watchObservedRunningTime="2026-10-30 21:38:51.521247887 +0000 UTC m=+1283.423111952" Oct 30 21:38:51.630170 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:51.630134 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:52.631861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:52.631825 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:38:52.632283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:52.631912 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:38:53.630178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:53.630139 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:38:53.630459 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:53.630204 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:38:54.629795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:54.629763 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:38:54.630210 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:54.629820 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:38:55.629560 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:55.629531 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:38:55.629748 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:55.629578 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:38:56.630374 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:56.630338 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:38:56.630862 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:56.630396 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:38:57.629495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:57.629442 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 21:38:57.629651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:57.629522 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 21:38:58.634252 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:58.634218 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:58.634661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:58.634587 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:38:58.635354 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:38:58.635332 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:39:02.432648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.432611 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv"] Oct 30 21:39:02.437010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.436984 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486403 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486354 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486403 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486411 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486504 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486534 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486560 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486746 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486642 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486743 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486772 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8v22\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.486928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.486868 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.541625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.541589 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv"] Oct 30 21:39:02.588348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588312 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588408 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588464 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588500 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8v22\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588581 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588652 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588705 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588759 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588787 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.588857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588804 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.589014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.588867 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.589057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.589020 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.589142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.589117 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.589361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.589337 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.591385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.591356 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.591495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.591466 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.645634 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.645602 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.645796 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.645671 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8v22\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:02.754763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:02.754671 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:03.183369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.183337 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv"] Oct 30 21:39:03.184318 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:39:03.184291 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e96d80_8fec_4476_9a30_700da90abc8d.slice/crio-02aad4986e5b937cf4c1d9419e1996056c89a090a86d0c4dc125a8e7afa3751b WatchSource:0}: Error finding container 02aad4986e5b937cf4c1d9419e1996056c89a090a86d0c4dc125a8e7afa3751b: Status 404 returned error can't find the container with id 02aad4986e5b937cf4c1d9419e1996056c89a090a86d0c4dc125a8e7afa3751b Oct 30 21:39:03.187782 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.187750 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:03.187904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.187841 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:03.187974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.187905 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:03.482393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.482306 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" event={"ID":"45e96d80-8fec-4476-9a30-700da90abc8d","Type":"ContainerStarted","Data":"8ac41409ca9f5f797a907f9cc09d33024e601361c7d33b573f3e72525cffad91"} Oct 30 21:39:03.482393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.482355 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" event={"ID":"45e96d80-8fec-4476-9a30-700da90abc8d","Type":"ContainerStarted","Data":"02aad4986e5b937cf4c1d9419e1996056c89a090a86d0c4dc125a8e7afa3751b"} Oct 30 21:39:03.606995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.606936 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podStartSLOduration=1.606915485 podStartE2EDuration="1.606915485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:03.604149074 +0000 UTC m=+1295.506013139" watchObservedRunningTime="2026-10-30 21:39:03.606915485 +0000 UTC m=+1295.508779550" Oct 30 21:39:03.755740 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:03.755645 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:04.756936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:04.756895 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:04.757362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:04.756953 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:05.755001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:05.754961 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:05.755188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:05.755026 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:06.755532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:06.755496 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:06.755941 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:06.755564 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:07.755658 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:07.755620 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:07.756071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:07.755688 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:08.299263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.299221 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l"] Oct 30 21:39:08.304764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.304733 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.385009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.384969 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.385327 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.385300 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.385460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.385444 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.385593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.385579 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92v8\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.392062 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.392032 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.392333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.392314 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.392484 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.392465 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.392756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.392739 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.392919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.392866 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.412897 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.412847 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l"] Oct 30 21:39:08.498857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.498807 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.498857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.498865 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.498961 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n92v8\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499049 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499104 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499135 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499248 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499274 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.499467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499428 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.500142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499822 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.500142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499844 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.500142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.499856 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.500142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.500101 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.500568 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.500541 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.502121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.502097 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.503649 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.503630 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.556916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.556824 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92v8\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.557130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.557102 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.619396 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.619355 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:08.756145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.755869 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:08.756145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:08.755974 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:09.095910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.092042 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l"] Oct 30 21:39:09.101330 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.101296 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:09.101482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.101377 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:09.101482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.101423 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:09.545643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.545608 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" event={"ID":"7a052d83-9021-4ac6-80ea-8b9e0473d3cf","Type":"ContainerStarted","Data":"1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c"} Oct 30 21:39:09.545833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.545664 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" event={"ID":"7a052d83-9021-4ac6-80ea-8b9e0473d3cf","Type":"ContainerStarted","Data":"56c08ef5c7d8502f61d1b28196faff42c5d76fff5e8d4165a07906c657aa1307"} Oct 30 21:39:09.620070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.620038 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:09.666120 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.666064 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podStartSLOduration=1.666048494 podStartE2EDuration="1.666048494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:09.663664806 +0000 UTC m=+1301.565528872" watchObservedRunningTime="2026-10-30 21:39:09.666048494 +0000 UTC m=+1301.567912559" Oct 30 21:39:09.755820 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.755783 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:39:09.756016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:09.755842 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:39:10.621190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:10.621154 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:10.621588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:10.621236 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:10.760134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:10.760103 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:10.760630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:10.760608 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:10.761385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:10.761367 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:39:11.620509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:11.620472 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:11.620724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:11.620573 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:12.345796 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:39:12.345752 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:39:12.620348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:12.620259 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:12.620348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:12.620310 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:12.801139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:12.801102 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:39:13.620628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:13.620590 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:13.621068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:13.620643 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:14.627778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:14.627305 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:14.627778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:14.627373 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:15.053102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:15.052979 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 21:39:15.630461 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:15.630423 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:15.631106 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:15.631070 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:15.758011 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:15.757965 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:15.918178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:15.916982 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.062078 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.062032 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.062928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.062904 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:39:16.086068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.086019 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.110910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.109971 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.198490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.198392 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:16.419905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.416788 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:39:16.426401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.426347 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:39:16.633917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.625744 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:16.633917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.625816 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:16.772681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.772636 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:17.005542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.806688 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 21:39:17.005850 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.848788 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.022100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:16.967616 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.063144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.063099 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 21:39:17.091983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.077975 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:39:17.091983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.078065 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:39:17.092658 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.092612 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.093267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.093245 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:39:17.153096 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.153050 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.155186 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.155158 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:39:17.215030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.199939 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.309966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.308314 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:39:17.542028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.541980 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.752239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.544629 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.754346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.754317 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:39:17.754579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.754564 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:39:17.755724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.567305 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:17.786864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.783684 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.811910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.677354 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 21:39:17.811910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.676210 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:17.811910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.803335 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:17.811910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.706996 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:17.811910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.803386 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:17.872047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:17.871997 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:18.006915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.006021 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:18.031921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.009001 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:39:18.031921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.010459 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:18.031921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.031123 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:39:18.065157 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.065111 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 21:39:18.111909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.090979 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:18.111909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.091047 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:18.111909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.092967 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:39:18.333963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.312294 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:18.333963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.312660 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:18.656895 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.656478 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:39:18.657446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.657404 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:39:18.755649 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.739283 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:39:18.820976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.820941 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:39:18.821268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.821250 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:39:18.821437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.821423 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:39:18.824626 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.824597 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:39:18.825374 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.825342 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:39:18.826346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.826325 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:39:18.828680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.828658 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:39:18.836385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.836360 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:39:18.841005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:18.840985 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:39:20.620953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:20.620909 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:20.621494 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:20.620978 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:20.721212 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:20.721175 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:20.722192 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:20.722164 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:20.723241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:20.723215 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:39:23.395508 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.395470 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz"] Oct 30 21:39:23.400417 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.400394 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.465626 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.465595 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz"] Oct 30 21:39:23.493009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.492972 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493077 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493121 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493206 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493328 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgzm\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493364 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493450 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493560 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.493644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.493614 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595324 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgzm\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595374 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595427 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595534 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595588 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595718 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595746 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595772 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.595811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.595803 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.596196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.596100 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.596196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.596162 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.596398 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.596376 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.596707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.596683 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.596841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.596815 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.598685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.598661 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.598801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.598722 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.652736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.652666 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.652868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.652815 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgzm\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm\") pod \"e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:23.715634 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:23.715594 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:24.155497 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.155471 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz"] Oct 30 21:39:24.156507 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:39:24.156484 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b53dc9_b989_4d3d_9c2f_15ce03381d63.slice/crio-10e1e9f817aad48e51501873d90da6c1d6a5213ae2c961589e3be5f177b1be0f WatchSource:0}: Error finding container 10e1e9f817aad48e51501873d90da6c1d6a5213ae2c961589e3be5f177b1be0f: Status 404 returned error can't find the container with id 10e1e9f817aad48e51501873d90da6c1d6a5213ae2c961589e3be5f177b1be0f Oct 30 21:39:24.158960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.158928 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:24.159060 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.158994 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:24.159060 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.159021 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:24.680317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.680277 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" event={"ID":"83b53dc9-b989-4d3d-9c2f-15ce03381d63","Type":"ContainerStarted","Data":"17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef"} Oct 30 21:39:24.680717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.680338 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" event={"ID":"83b53dc9-b989-4d3d-9c2f-15ce03381d63","Type":"ContainerStarted","Data":"10e1e9f817aad48e51501873d90da6c1d6a5213ae2c961589e3be5f177b1be0f"} Oct 30 21:39:24.715737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:24.715707 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:25.718386 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:25.718342 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:39:25.718812 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:25.718428 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:39:26.716966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:26.716933 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:39:26.717169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:26.716989 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:39:27.716547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:27.716510 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:39:27.717001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:27.716580 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:39:28.716199 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:28.716155 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:39:28.716417 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:28.716214 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:39:29.716273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:29.716231 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:39:29.716675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:29.716290 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:39:30.716186 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:30.716152 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:39:30.716356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:30.716217 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:39:31.720572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:31.720541 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:31.721037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:31.720969 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:31.721663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:31.721644 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:39:31.889273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:31.889221 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podStartSLOduration=8.88920721 podStartE2EDuration="8.88920721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:24.849094846 +0000 UTC m=+1316.750958911" watchObservedRunningTime="2026-10-30 21:39:31.88920721 +0000 UTC m=+1323.791071344" Oct 30 21:39:35.652452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.652402 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh"] Oct 30 21:39:35.657943 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.657916 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.703590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703543 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.703809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703597 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.703809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703730 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.703809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703768 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.703809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703793 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.704004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703814 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.704004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.703952 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.704071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.704008 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.704172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.704151 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.714865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.714830 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh"] Oct 30 21:39:35.805090 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805054 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805144 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805172 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805218 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805261 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805291 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805315 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805360 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805431 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805481 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805590 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805876 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.805979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.805929 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.806184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.806123 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.807801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.807780 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.807947 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.807930 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.862257 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.862228 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.862407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.862274 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5\") pod \"e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:35.972120 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:35.972028 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:36.406662 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.406627 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh"] Oct 30 21:39:36.422234 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.422182 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:36.422400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.422284 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:36.422400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.422325 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:36.777205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.777119 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" event={"ID":"7dc5d51e-5931-4fdf-ac4f-aaff7658b730","Type":"ContainerStarted","Data":"f304e027cad8b0dca0643be78cd94226ee3febd3c05fc29c6168f8e2772a3e49"} Oct 30 21:39:36.777205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.777176 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" event={"ID":"7dc5d51e-5931-4fdf-ac4f-aaff7658b730","Type":"ContainerStarted","Data":"95fa03211eab2ed1c8f7ef847fcf2b831c591a38253db04915a9d6473607b152"} Oct 30 21:39:36.973005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:36.972967 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:37.993908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:37.984044 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:37.993908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:37.984114 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:38.806824 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.806757 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podStartSLOduration=3.806737699 podStartE2EDuration="3.806737699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:36.895848699 +0000 UTC m=+1328.797712768" watchObservedRunningTime="2026-10-30 21:39:38.806737699 +0000 UTC m=+1330.708601766" Oct 30 21:39:38.809749 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.809712 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5"] Oct 30 21:39:38.823596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.818143 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848188 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848427 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848249 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848427 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848276 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7l8v\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848454 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848511 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848645 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848682 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848715 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.848823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.848766 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.872069 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.872036 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5"] Oct 30 21:39:38.950325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950286 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950334 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950359 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7l8v\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950422 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950441 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950569 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950595 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950630 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950657 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.950970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.950864 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.951119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.951091 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.951266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.951244 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.951335 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.951286 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.951539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.951512 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.953145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.953115 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.953585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.953564 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:38.972654 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.972623 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:38.972811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:38.972678 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:39.008525 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.008485 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:39.008899 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.008606 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7l8v\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:39.107023 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.106926 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc"] Oct 30 21:39:39.112577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.112542 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7"] Oct 30 21:39:39.112741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.112661 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.117273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.117252 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.141439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.141407 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:39.158044 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.157992 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.158331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158048 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.158331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158105 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.158331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158146 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.158331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158252 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.158686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158388 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.158686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158455 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.158686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158511 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptq9g\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.158686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158542 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.158686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158598 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.158686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158648 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158689 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158730 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158796 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84x4d\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158869 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158920 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.158979 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.159100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.159015 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.166545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.166510 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc"] Oct 30 21:39:39.208836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.208237 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7"] Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262625 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262680 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262746 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262781 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262873 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262926 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.262961 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263001 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263056 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263100 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263169 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263245 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptq9g\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263276 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263342 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263397 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263432 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263472 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.263556 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84x4d\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.264813 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.265339 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.265873 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.266370 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.267592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.267310 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.269341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.267848 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.269341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.267866 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.269341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.268033 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.269341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.268111 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.269341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.268605 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.269600 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.269452 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.279159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.278246 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.279159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.278646 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.305177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.302567 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.320925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.320835 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.321454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.321390 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84x4d\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.324760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.324726 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptq9g\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.326262 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.326234 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token\") pod \"e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.428673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.428637 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:39.436306 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.436283 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:39.618965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.618532 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5"] Oct 30 21:39:39.623375 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:39:39.623340 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5bd682_1e8c_45d2_bda9_5d8a6ea96019.slice/crio-bac429499c061e0d19cc5457aa9de982c9cc36f427d87b51b4fa8bada4d6c848 WatchSource:0}: Error finding container bac429499c061e0d19cc5457aa9de982c9cc36f427d87b51b4fa8bada4d6c848: Status 404 returned error can't find the container with id bac429499c061e0d19cc5457aa9de982c9cc36f427d87b51b4fa8bada4d6c848 Oct 30 21:39:39.629928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.626712 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.629928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.626779 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.629928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.626811 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.819167 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.819130 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" event={"ID":"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019","Type":"ContainerStarted","Data":"72dbe157cef1ad9e0c84e8ec9e60dca1a4e25975337727246b3d46b0b750462d"} Oct 30 21:39:39.819345 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.819194 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" event={"ID":"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019","Type":"ContainerStarted","Data":"bac429499c061e0d19cc5457aa9de982c9cc36f427d87b51b4fa8bada4d6c848"} Oct 30 21:39:39.843864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.843825 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc"] Oct 30 21:39:39.845256 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:39:39.845231 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502d78a3_3a23_4a04_87f0_89aa189dd010.slice/crio-a64abb0311eac56937b518ecdcc0847b5802fe679547a4d97333368891418d52 WatchSource:0}: Error finding container a64abb0311eac56937b518ecdcc0847b5802fe679547a4d97333368891418d52: Status 404 returned error can't find the container with id a64abb0311eac56937b518ecdcc0847b5802fe679547a4d97333368891418d52 Oct 30 21:39:39.847982 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.847947 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.848084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.848011 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.848084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.848039 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.907012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.906963 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7"] Oct 30 21:39:39.907367 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:39:39.907338 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0900c473_515d_44c1_8ce6_06d8e9f4d350.slice/crio-2eb5f510757a8c30423cfbbce2e2777e9a93cfadd8a5d4f60858bf79f70edd72 WatchSource:0}: Error finding container 2eb5f510757a8c30423cfbbce2e2777e9a93cfadd8a5d4f60858bf79f70edd72: Status 404 returned error can't find the container with id 2eb5f510757a8c30423cfbbce2e2777e9a93cfadd8a5d4f60858bf79f70edd72 Oct 30 21:39:39.911013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.910874 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.911013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.910988 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.911121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.911037 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:39:39.939557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.939510 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podStartSLOduration=1.939496096 podStartE2EDuration="1.939496096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:39.937622352 +0000 UTC m=+1331.839486416" watchObservedRunningTime="2026-10-30 21:39:39.939496096 +0000 UTC m=+1331.841360161" Oct 30 21:39:39.972962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.972925 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:39.973120 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:39.972984 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:40.142672 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:40.142630 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:40.144555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:40.144527 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:40.144705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:40.144579 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:41.358908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.358078 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:41.358908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.358326 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:41.468263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.467001 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:41.481641 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.467069 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:41.661442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.661391 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:41.867930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.822852 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:41.893907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.843178 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:41.895832 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.848975 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:39:41.897093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:41.885000 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.097018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.096440 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.097018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.096597 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:42.163755 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.163714 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" event={"ID":"0900c473-515d-44c1-8ce6-06d8e9f4d350","Type":"ContainerStarted","Data":"e83b3d0ea486304f6787b4dbafec18c16b48c7bba7331497992db48bfe6363a3"} Oct 30 21:39:42.164103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.164079 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" event={"ID":"0900c473-515d-44c1-8ce6-06d8e9f4d350","Type":"ContainerStarted","Data":"2eb5f510757a8c30423cfbbce2e2777e9a93cfadd8a5d4f60858bf79f70edd72"} Oct 30 21:39:42.174095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.174063 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.174343 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.174319 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:42.174972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.174935 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" event={"ID":"502d78a3-3a23-4a04-87f0-89aa189dd010","Type":"ContainerStarted","Data":"01f7786027aec1d1333f97ea40671b4d741c808ba713583937a2465675510c60"} Oct 30 21:39:42.175097 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.175002 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" event={"ID":"502d78a3-3a23-4a04-87f0-89aa189dd010","Type":"ContainerStarted","Data":"a64abb0311eac56937b518ecdcc0847b5802fe679547a4d97333368891418d52"} Oct 30 21:39:42.190918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.189974 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.191229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.191195 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.240021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.239975 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.517970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.487533 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:42.517970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.487633 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.519471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.518727 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:42.519471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.519353 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.519471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.519404 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:39:42.567037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.566969 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podStartSLOduration=3.566946272 podStartE2EDuration="3.566946272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:42.348406327 +0000 UTC m=+1334.250270392" watchObservedRunningTime="2026-10-30 21:39:42.566946272 +0000 UTC m=+1334.468810338" Oct 30 21:39:42.644908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.635423 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podStartSLOduration=3.635403187 podStartE2EDuration="3.635403187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:42.613168868 +0000 UTC m=+1334.515032934" watchObservedRunningTime="2026-10-30 21:39:42.635403187 +0000 UTC m=+1334.537267257" Oct 30 21:39:42.673923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.661305 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:39:42.673923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.661363 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:39:42.673923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.666693 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.673923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.667138 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:39:42.806977 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:39:42.803499 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:39:42.871909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.868445 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.901907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.900079 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.901907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.900483 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:39:42.902175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.902057 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:42.947917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.902327 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:39:42.947917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.902603 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:39:42.947917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:42.902655 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:39:43.069141 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.069050 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.069408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.069374 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:43.106331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.106288 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.115766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.115420 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:39:43.147527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.147238 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.171710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.171668 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:43.228907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.225404 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.281643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.253061 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.290945 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.290079 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:39:43.291229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.291210 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:39:43.363266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.363167 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:39:43.490761 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.490726 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.898674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.898625 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:39:43.899357 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.670976 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.899811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.871757 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.900312 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.669998 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.900568 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.900544 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:39:43.901642 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.491460 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.901804 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.901780 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:39:43.906911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.906711 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:43.907034 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.907008 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:43.983014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.982980 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:39:43.983283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:43.983246 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:39:44.133436 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.133376 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:44.195968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.193044 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:44.195968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.193106 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:44.297017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.296976 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:44.301016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.300958 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:44.542983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.537499 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:39:44.542983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.537560 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:39:44.542983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.537643 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:39:44.542983 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:44.537664 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:39:45.064908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.053006 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:45.103228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.103179 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:45.121907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.113775 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:45.121907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.114081 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:39:45.137246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.137205 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:39:45.159631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.159587 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:45.178060 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.178013 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:45.318132 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.316991 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 21:39:45.318132 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.316991 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:39:45.438909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.434572 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:39:45.438909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.434637 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:39:45.457937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.439682 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:39:45.457937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.439736 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:39:45.484907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.458634 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:45.484907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.460261 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:45.527917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.487499 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:39:45.563251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.563217 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:39:45.568212 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.568136 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:39:45.572166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.572145 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:39:45.672592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.672563 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:39:45.758543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.758341 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:39:45.760290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.760262 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:39:45.767922 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:45.767904 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:39:46.166855 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:46.166807 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.167493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:46.167450 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:46.453957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:46.452265 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.453957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:46.452326 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:39:46.453957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:46.452631 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:39:46.453957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:46.452667 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:39:47.142226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:47.142191 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:39:47.142491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:47.142453 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:39:47.429561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:47.429458 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:39:47.429561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:47.429532 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 21:39:47.436908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:47.436861 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:39:47.437001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:47.436937 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 21:39:48.604013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.603611 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:48.605261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.605230 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:48.606549 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.606499 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 21:39:48.655258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.655223 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:48.655779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.655717 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:48.658620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.658598 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:39:48.770588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.770555 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:48.771687 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.771661 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:48.772491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:48.772469 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:39:50.764003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:50.763957 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc"] Oct 30 21:39:50.764542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:50.764188 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" containerID="cri-o://3c4179e6699bc899d75a9261992dc70ee50178f9d51a1dc685912533de2113f4" gracePeriod=30 Oct 30 21:39:55.913939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:55.911749 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:55.913939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:55.912634 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:55.913939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:55.912925 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.023905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.018977 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.151956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.148034 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:39:56.464391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.455849 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.464391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.461585 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:39:56.646258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.631802 5914 generic.go:356] "Generic (PLEG): container finished" podID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerID="3c4179e6699bc899d75a9261992dc70ee50178f9d51a1dc685912533de2113f4" exitCode=0 Oct 30 21:39:56.674020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.673974 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.781969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.778192 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" event={"ID":"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f","Type":"ContainerDied","Data":"3c4179e6699bc899d75a9261992dc70ee50178f9d51a1dc685912533de2113f4"} Oct 30 21:39:56.783381 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.783343 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.814121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.813993 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 21:39:56.819919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.819217 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:56.819919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.819302 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:56.942908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.942496 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:56.962229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.943802 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:56.962229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.944860 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:39:56.962229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.945203 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:39:56.962229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.945263 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:39:56.962229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:56.945299 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:39:57.026137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.019173 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.026137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.019687 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:39:57.185000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.183976 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.225924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.196949 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:39:57.505965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.505103 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:39:57.505965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.505528 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:39:57.532908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.508251 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:39:57.532908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.508576 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:39:57.634046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.633999 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.780963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.774018 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.798911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.795622 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:39:57.873381 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.873332 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.917910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.899591 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:39:57.983621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.983568 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:57.997915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.986340 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:58.016914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:57.986830 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:58.083947 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.003950 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 21:39:58.086036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.033631 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:39:58.086294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.033675 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:39:58.137070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137036 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.137345 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137325 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.137544 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137528 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.137680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137666 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.137829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137816 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.137965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137949 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.138195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.138181 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.138363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.138349 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.138479 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.138467 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.170935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.156133 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:58.170935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.158118 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:58.170935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.137182 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" event={"ID":"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f","Type":"ContainerDied","Data":"7569ddc6dd7ccaf5321e476a9b24e9f78950836255fac19b6db2885d3a26f0f1"} Oct 30 21:39:58.170935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.158560 5914 scope.go:122] "RemoveContainer" containerID="3c4179e6699bc899d75a9261992dc70ee50178f9d51a1dc685912533de2113f4" Oct 30 21:39:58.224949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.185380 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:39:58.224949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.195683 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:58.224949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.196153 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:58.224949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.197035 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:58.224949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.197983 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:58.224949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.218437 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327843 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327900 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327918 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327933 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327946 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327959 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.329372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.327971 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.431414 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.414928 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:39:58.459450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.459409 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "kube-api-access-w5q4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:58.651964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.564472 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:39:58.651964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.564874 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:39:58.772912 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.753945 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\") pod \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\" (UID: \"c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f\") " Oct 30 21:39:58.910230 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.818594 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.910230 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:39:58.769411 5914 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f/volumes/kubernetes.io~projected/kube-api-access-w5q4n Oct 30 21:39:58.910230 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.818694 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n" pod "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" (UID: "c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f"). InnerVolumeSpecName "kube-api-access-w5q4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:58.910230 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.832049 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:39:58.915990 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.915949 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:58.930358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.930328 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5q4n\" (UniqueName: \"kubernetes.io/projected/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f-kube-api-access-w5q4n\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:39:58.957976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:58.946919 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 21:39:59.050925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.037685 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:39:59.074355 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.057072 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.102960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.089347 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:39:59.288976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.266087 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.459333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.362058 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:39:59.597971 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.591482 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:39:59.597971 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.591913 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:39:59.702028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.701985 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:59.702318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.702289 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.815349 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.815318 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc" Oct 30 21:39:59.838907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.835776 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.869960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.866114 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:39:59.870540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.870500 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.914908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.911130 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:39:59.922795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:39:59.922750 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.043119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.043066 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:40:00.102783 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.101217 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:40:00.112018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.104378 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:40:00.611322 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.611276 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:00.614318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.614288 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc"] Oct 30 21:40:00.614551 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.614534 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjrxc"] Oct 30 21:40:00.614705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.614692 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:40:00.614822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.614811 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:40:00.614957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.614945 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:40:00.615100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.615086 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:40:00.615254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.615242 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:40:00.615389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.615376 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:40:00.615510 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.615498 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:40:00.728656 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.728624 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:40:00.818956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:00.818915 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:40:02.219846 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:02.219809 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" path="/var/lib/kubelet/pods/c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f/volumes" Oct 30 21:40:03.283932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.283860 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x"] Oct 30 21:40:03.284674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.284656 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" Oct 30 21:40:03.284717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.284676 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" Oct 30 21:40:03.284810 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.284801 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1049aab-2a9b-4c8b-ac8b-83a2b7b0d24f" containerName="istio-proxy" Oct 30 21:40:03.289573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.289555 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.347748 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.347708 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x"] Oct 30 21:40:03.376264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376227 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376276 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376510 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376458 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376506 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376629 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376679 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376735 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wl8\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376830 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.376993 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.376967 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478242 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478204 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478266 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478294 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478372 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478399 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478446 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478485 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478524 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wl8\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478669 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.478685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.478679 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.479204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.479150 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.479204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.479154 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.479204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.479193 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.479644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.479618 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.480989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.480952 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.481459 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.481431 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.545216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.544702 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.547342 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.547313 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wl8\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:03.604317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:03.604285 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:04.034948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:04.034915 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x"] Oct 30 21:40:04.036106 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:04.036069 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31752b73_efb9_4aa6_972b_544e87208bd9.slice/crio-26afec28683a9552efe966bf870a02d3f1b5b747c9cabb223105932ef17a4c25 WatchSource:0}: Error finding container 26afec28683a9552efe966bf870a02d3f1b5b747c9cabb223105932ef17a4c25: Status 404 returned error can't find the container with id 26afec28683a9552efe966bf870a02d3f1b5b747c9cabb223105932ef17a4c25 Oct 30 21:40:04.039110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:04.039078 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:04.039214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:04.039150 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:04.039214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:04.039182 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:04.855097 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:04.855067 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" event={"ID":"31752b73-efb9-4aa6-972b-544e87208bd9","Type":"ContainerStarted","Data":"469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039"} Oct 30 21:40:04.855516 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:04.855118 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" event={"ID":"31752b73-efb9-4aa6-972b-544e87208bd9","Type":"ContainerStarted","Data":"26afec28683a9552efe966bf870a02d3f1b5b747c9cabb223105932ef17a4c25"} Oct 30 21:40:05.072026 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:05.071968 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podStartSLOduration=2.071951534 podStartE2EDuration="2.071951534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:05.021092837 +0000 UTC m=+1356.922956903" watchObservedRunningTime="2026-10-30 21:40:05.071951534 +0000 UTC m=+1356.973815599" Oct 30 21:40:05.605948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:05.605907 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:05.608156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:05.608128 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:05.608328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:05.608185 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:06.605592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:06.605502 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:06.606083 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:06.605601 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:07.606066 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:07.606022 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:07.606484 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:07.606087 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:08.607273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.607235 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:08.607920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.607855 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:08.628755 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.628716 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn"] Oct 30 21:40:08.689503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.689452 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn"] Oct 30 21:40:08.710997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.700132 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831232 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831455 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831501 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831539 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831578 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831615 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831663 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94gms\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831736 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.842959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.831780 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.969628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.891586 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d56d1199-gw-openshift-default-dockercfg-rhkkh\"" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.971496 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94gms\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.971588 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.971645 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.971723 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.972008 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.972054 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.972094 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.972131 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:08.980965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:08.972165 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.071574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.071536 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.072933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.072231 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.072933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.072723 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.073131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.073014 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.080904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.073252 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.115663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.114413 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.115663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.114731 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.150575 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.150536 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.217908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.211893 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94gms\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms\") pod \"e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.421955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.419966 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:09.543403 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.543339 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc"] Oct 30 21:40:09.551267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.551238 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.606908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.606260 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bed6793d-openshift-default-dockercfg-6ldgq\"" Oct 30 21:40:09.619901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.619857 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc"] Oct 30 21:40:09.620267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.620237 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:09.620322 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.620296 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:09.648367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648257 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648299 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648436 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648498 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648527 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648585 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648673 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.648953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648934 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.649010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.648977 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncwb\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.751224 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.751129 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.823410 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.803162 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.823752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.823723 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncwb\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.848379 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.848452 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.848522 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.848584 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.848715 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.848784 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.849428 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.849906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.777946 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.850514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.850158 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.850514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.805844 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.850514 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.850404 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.850686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.850609 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:09.894000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:09.886027 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:10.009167 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.005368 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncwb\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:10.030439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.030393 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:10.030700 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.030676 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:10.030851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.030833 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:10.057485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.057303 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn"] Oct 30 21:40:10.059122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.057717 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-z44vc\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:10.105008 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.104932 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" event={"ID":"ac1ee904-e6de-4aa6-abd9-279746863133","Type":"ContainerStarted","Data":"9a6a36ea22961a1cfb82ffe793c89dd6ed76416c305a1fc71f901f2671243173"} Oct 30 21:40:10.172341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.172032 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:10.605289 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.605200 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:40:10.605578 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.605535 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:40:10.643920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.643868 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc"] Oct 30 21:40:10.645457 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:10.645430 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1bae9c_1080_472d_b688_100c93cef310.slice/crio-15247a86fde505d3a19ab3192100f82353e26a2ac56c2c6bf68c38de48192970 WatchSource:0}: Error finding container 15247a86fde505d3a19ab3192100f82353e26a2ac56c2c6bf68c38de48192970: Status 404 returned error can't find the container with id 15247a86fde505d3a19ab3192100f82353e26a2ac56c2c6bf68c38de48192970 Oct 30 21:40:10.649103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.649064 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:10.649199 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.649159 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:10.649244 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:10.649202 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:11.117246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.117202 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" event={"ID":"ac1ee904-e6de-4aa6-abd9-279746863133","Type":"ContainerStarted","Data":"ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d"} Oct 30 21:40:11.119642 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.119609 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" event={"ID":"ff1bae9c-1080-472d-b688-100c93cef310","Type":"ContainerStarted","Data":"dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed"} Oct 30 21:40:11.119760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.119670 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" event={"ID":"ff1bae9c-1080-472d-b688-100c93cef310","Type":"ContainerStarted","Data":"15247a86fde505d3a19ab3192100f82353e26a2ac56c2c6bf68c38de48192970"} Oct 30 21:40:11.173227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.173191 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:11.178644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.178618 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:11.239469 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.239343 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podStartSLOduration=3.239322716 podStartE2EDuration="3.239322716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:11.23479736 +0000 UTC m=+1363.136661431" watchObservedRunningTime="2026-10-30 21:40:11.239322716 +0000 UTC m=+1363.141186783" Oct 30 21:40:11.351726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.351658 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podStartSLOduration=2.351637656 podStartE2EDuration="2.351637656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:11.350280556 +0000 UTC m=+1363.252144622" watchObservedRunningTime="2026-10-30 21:40:11.351637656 +0000 UTC m=+1363.253501722" Oct 30 21:40:11.427145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.427109 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:11.433324 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:11.433291 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:12.127289 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.127084 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:12.127289 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.127163 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:12.129197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.128610 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:40:12.129197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.128681 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:40:12.138972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.138943 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:12.139304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.139283 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:12.140346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:12.140322 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:40:12.347194 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:40:12.347084 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:40:17.671577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.671537 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk"] Oct 30 21:40:17.676183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.676158 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:17.734906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.730027 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d56d1199\"" Oct 30 21:40:17.772165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.772109 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:17.772565 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.772539 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jqcd\" (UniqueName: \"kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd\") pod \"payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:17.792101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.792067 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d56d1199-dockercfg-r9vsb\"" Oct 30 21:40:17.815532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.815496 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk"] Oct 30 21:40:17.880762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.879084 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:17.880762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.879250 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jqcd\" (UniqueName: \"kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd\") pod \"payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:17.880762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.880707 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:17.899790 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:17.899755 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n"] Oct 30 21:40:18.021045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.020865 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n"] Oct 30 21:40:18.021277 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.021013 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.038045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.033076 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jqcd\" (UniqueName: \"kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd\") pod \"payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:18.084638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.084601 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h"] Oct 30 21:40:18.089389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.089359 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.113656 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.113623 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.113820 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.113688 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4q4\" (UniqueName: \"kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4\") pod \"payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.146786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.146750 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h"] Oct 30 21:40:18.215370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215342 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215407 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215489 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215519 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215549 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215748 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215726 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.215795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215759 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215806 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.215930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.215906 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.216052 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.216032 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4q4\" (UniqueName: \"kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4\") pod \"payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.216129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.216089 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8znj\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.216399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.216381 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.272968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.272849 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4q4\" (UniqueName: \"kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4\") pod \"payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.293313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.293283 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:18.317914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.317855 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318078 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.317960 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318078 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.317980 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318078 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318064 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8znj\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318198 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318262 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318328 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318368 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318386 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318415 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318497 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318638 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.318954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.318848 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.319048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.319030 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.320816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.320782 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.321208 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.321183 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.356368 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.356337 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:18.377586 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.377233 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8znj\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.380513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.380455 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.406015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.405730 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:18.672295 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.672265 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk"] Oct 30 21:40:18.674285 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:18.674258 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb379de_f7c9_45fe_84e5_3211b6499794.slice/crio-ffe5800676f086078e3b9ad76a02ed17b68d89644a3c8f896191d8180fd8e5ab WatchSource:0}: Error finding container ffe5800676f086078e3b9ad76a02ed17b68d89644a3c8f896191d8180fd8e5ab: Status 404 returned error can't find the container with id ffe5800676f086078e3b9ad76a02ed17b68d89644a3c8f896191d8180fd8e5ab Oct 30 21:40:18.764667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.764638 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n"] Oct 30 21:40:18.765607 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:18.765583 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c44cc1_e025_4ab0_8691_bf42d4c846c8.slice/crio-55595e93cd05263992d595aa9d5a4eb2b202304d505f10550e57b392e4ff7b8d WatchSource:0}: Error finding container 55595e93cd05263992d595aa9d5a4eb2b202304d505f10550e57b392e4ff7b8d: Status 404 returned error can't find the container with id 55595e93cd05263992d595aa9d5a4eb2b202304d505f10550e57b392e4ff7b8d Oct 30 21:40:18.808162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.808129 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h"] Oct 30 21:40:18.824461 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:18.824424 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f822d9_ca18_4779_a370_a289ba734096.slice/crio-2003cafaaa0a7aa47909e08523a690b8a84bb573b624b3d987e03257cafe3480 WatchSource:0}: Error finding container 2003cafaaa0a7aa47909e08523a690b8a84bb573b624b3d987e03257cafe3480: Status 404 returned error can't find the container with id 2003cafaaa0a7aa47909e08523a690b8a84bb573b624b3d987e03257cafe3480 Oct 30 21:40:18.827494 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.827456 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:18.827599 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.827538 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:18.827599 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:18.827578 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:19.186006 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.185956 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" event={"ID":"a7c44cc1-e025-4ab0-8691-bf42d4c846c8","Type":"ContainerStarted","Data":"1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3"} Oct 30 21:40:19.186209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.186051 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" event={"ID":"a7c44cc1-e025-4ab0-8691-bf42d4c846c8","Type":"ContainerStarted","Data":"55595e93cd05263992d595aa9d5a4eb2b202304d505f10550e57b392e4ff7b8d"} Oct 30 21:40:19.186209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.186188 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:19.188308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.188278 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" event={"ID":"89f822d9-ca18-4779-a370-a289ba734096","Type":"ContainerStarted","Data":"9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d"} Oct 30 21:40:19.188437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.188330 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" event={"ID":"89f822d9-ca18-4779-a370-a289ba734096","Type":"ContainerStarted","Data":"2003cafaaa0a7aa47909e08523a690b8a84bb573b624b3d987e03257cafe3480"} Oct 30 21:40:19.190099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.190074 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" event={"ID":"ecb379de-f7c9-45fe-84e5-3211b6499794","Type":"ContainerStarted","Data":"48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61"} Oct 30 21:40:19.190236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.190119 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" event={"ID":"ecb379de-f7c9-45fe-84e5-3211b6499794","Type":"ContainerStarted","Data":"ffe5800676f086078e3b9ad76a02ed17b68d89644a3c8f896191d8180fd8e5ab"} Oct 30 21:40:19.190236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.190154 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:19.309238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.309185 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" podStartSLOduration=2.309170487 podStartE2EDuration="2.309170487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.304996631 +0000 UTC m=+1371.206860699" watchObservedRunningTime="2026-10-30 21:40:19.309170487 +0000 UTC m=+1371.211034551" Oct 30 21:40:19.406955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.406910 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:19.408753 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.408730 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:19.408825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.408781 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:19.602968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:19.602157 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" podStartSLOduration=2.60211831 podStartE2EDuration="2.60211831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.440207071 +0000 UTC m=+1371.342071137" watchObservedRunningTime="2026-10-30 21:40:19.60211831 +0000 UTC m=+1371.503982376" Oct 30 21:40:20.407267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:20.407236 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:20.407651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:20.407290 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:21.406851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:21.406815 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:21.407078 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:21.406911 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:22.407402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:22.407350 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:22.407981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:22.407431 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:23.406614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:23.406575 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:23.406811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:23.406630 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:24.211064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.210996 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podStartSLOduration=6.210978075 podStartE2EDuration="6.210978075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:19.582150623 +0000 UTC m=+1371.484014691" watchObservedRunningTime="2026-10-30 21:40:24.210978075 +0000 UTC m=+1376.112842138" Oct 30 21:40:24.218934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.218908 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl"] Oct 30 21:40:24.223336 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.223314 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.350961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.350923 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351181 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl"] Oct 30 21:40:24.351391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351363 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351410 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351571 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351546 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351635 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351731 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351713 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351825 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351914 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7cx\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.351992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.351956 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.407326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.407290 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:24.407503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.407343 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:24.453378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453342 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453385 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453464 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453520 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453594 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453579 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453640 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453711 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7cx\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453757 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453806 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453829 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.453906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.453861 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.454939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.454869 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.455194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.455149 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.455303 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.455286 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.457553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.457533 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.458262 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.458228 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.513739 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.513672 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.513910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.513872 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7cx\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx\") pod \"e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:24.539288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:24.539258 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:25.368974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:25.368938 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl"] Oct 30 21:40:25.397559 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:25.397526 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcda770e_128b_454c_a843_38916d2f7603.slice/crio-55cff74f6aa03f9d4285d0775649ff64829a15d15b275ebbc3cc5b0fd90a2415 WatchSource:0}: Error finding container 55cff74f6aa03f9d4285d0775649ff64829a15d15b275ebbc3cc5b0fd90a2415: Status 404 returned error can't find the container with id 55cff74f6aa03f9d4285d0775649ff64829a15d15b275ebbc3cc5b0fd90a2415 Oct 30 21:40:25.401096 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:25.401059 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:25.401222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:25.401148 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:25.401222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:25.401199 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:40:25.407104 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:25.407082 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:40:25.407210 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:25.407131 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:40:26.423819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.423782 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" event={"ID":"bcda770e-128b-454c-a843-38916d2f7603","Type":"ContainerStarted","Data":"5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091"} Oct 30 21:40:26.428674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.428625 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" event={"ID":"bcda770e-128b-454c-a843-38916d2f7603","Type":"ContainerStarted","Data":"55cff74f6aa03f9d4285d0775649ff64829a15d15b275ebbc3cc5b0fd90a2415"} Oct 30 21:40:26.540077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.540043 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:26.541564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.541532 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:26.541670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.541602 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:26.662396 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.662340 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podStartSLOduration=2.662325889 podStartE2EDuration="2.662325889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:26.611596939 +0000 UTC m=+1378.513461003" watchObservedRunningTime="2026-10-30 21:40:26.662325889 +0000 UTC m=+1378.564189954" Oct 30 21:40:26.671630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.671190 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:26.671630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.671547 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:26.672865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:26.672839 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:40:27.555972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:27.541283 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:27.555972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:27.541391 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:27.724036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:27.724000 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2"] Oct 30 21:40:27.976611 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:27.964562 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:27.993202 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:27.973809 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2"] Oct 30 21:40:28.046251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.046219 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-385396af\"" Oct 30 21:40:28.046251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.046219 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-385396af-dockercfg-szntk\"" Oct 30 21:40:28.118192 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.118158 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrjd\" (UniqueName: \"kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd\") pod \"payload-processing-e2e-derive-385396af-64b8655bd5-tckp2\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.118383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.118340 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume\") pod \"payload-processing-e2e-derive-385396af-64b8655bd5-tckp2\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.219452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.219414 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrjd\" (UniqueName: \"kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd\") pod \"payload-processing-e2e-derive-385396af-64b8655bd5-tckp2\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.219631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.219613 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume\") pod \"payload-processing-e2e-derive-385396af-64b8655bd5-tckp2\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.270451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.270374 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-385396af\"" Oct 30 21:40:28.299309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.294002 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume\") pod \"payload-processing-e2e-derive-385396af-64b8655bd5-tckp2\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.300826 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.300788 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrjd\" (UniqueName: \"kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd\") pod \"payload-processing-e2e-derive-385396af-64b8655bd5-tckp2\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.398913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.390238 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-385396af-dockercfg-szntk\"" Oct 30 21:40:28.398913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.390317 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:28.540028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.539803 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:28.540028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.539919 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:28.822960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:28.814688 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2"] Oct 30 21:40:28.849834 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:40:28.849799 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8234fd05_22f4_4591_81f7_c0281ffa1e72.slice/crio-fea1cd9561484a3301f1e097f72cefb874c1dc8ad26767e5b3e7839a213baea9 WatchSource:0}: Error finding container fea1cd9561484a3301f1e097f72cefb874c1dc8ad26767e5b3e7839a213baea9: Status 404 returned error can't find the container with id fea1cd9561484a3301f1e097f72cefb874c1dc8ad26767e5b3e7839a213baea9 Oct 30 21:40:29.539753 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:29.539712 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:29.539976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:29.539769 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:29.814064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:29.813959 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" event={"ID":"8234fd05-22f4-4591-81f7-c0281ffa1e72","Type":"ContainerStarted","Data":"2bb44f9b3145b723cbfd3fed8c2f8f32b3d48a3cf5b0115777bbf99f7d20e96f"} Oct 30 21:40:29.814245 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:29.814058 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" event={"ID":"8234fd05-22f4-4591-81f7-c0281ffa1e72","Type":"ContainerStarted","Data":"fea1cd9561484a3301f1e097f72cefb874c1dc8ad26767e5b3e7839a213baea9"} Oct 30 21:40:29.814296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:29.814265 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:29.930050 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:29.929985 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" podStartSLOduration=2.929966636 podStartE2EDuration="2.929966636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:29.927577959 +0000 UTC m=+1381.829442023" watchObservedRunningTime="2026-10-30 21:40:29.929966636 +0000 UTC m=+1381.831830702" Oct 30 21:40:30.238522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:30.238478 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:40:30.238692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:30.238584 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:40:30.540501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:30.540410 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:30.540501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:30.540469 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:31.540530 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:31.540495 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:31.540998 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:31.540565 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:32.427732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:32.427699 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:40:32.539989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:32.539956 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:40:32.540167 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:32.540006 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:40:34.610114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:34.572083 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.610114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:34.599825 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:34.610114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:34.599907 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:34.818904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:34.815202 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.052048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.051997 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.089046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.088989 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.202909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.198815 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:40:35.612915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.588172 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:40:35.613873 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.593998 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:35.635252 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.594038 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:40:35.639908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.635860 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:40:35.738103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.700155 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:35.738435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.738394 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:35.823188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.823144 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.001139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.904990 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.001139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:35.961612 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.125909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.090980 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:40:36.142569 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.122074 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:40:36.195046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.193985 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:40:36.234908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.202662 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.234908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.212001 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:40:36.390955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.390031 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.655821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.655716 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:40:36.664909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.661631 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:36.692757 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.692717 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:40:36.763456 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.763418 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:36.763736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.763706 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:36.829936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.829830 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:40:36.829936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.829913 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:40:36.830213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.829948 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:40:36.927969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.852247 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.000621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:36.859308 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.144909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.143982 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.215967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.200063 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.215967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.200550 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:40:37.266932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.225739 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 21:40:37.266932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.226195 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:40:37.384107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.384057 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:40:37.398102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.398050 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:40:37.580176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.580074 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:37.634911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.604099 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:37.835421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.835323 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:40:38.003112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.983281 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:38.030558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.997302 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.031078 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.997348 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:40:38.043070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.998364 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.043330 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:37.998415 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 21:40:38.043460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.003036 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:38.043602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.043568 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.211643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.171412 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.212247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.201061 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.226668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.226626 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:38.463294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.456537 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:40:38.463294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.456580 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:40:38.944905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:38.937125 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:40:39.113976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.112203 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:40:39.184239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.184196 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 21:40:39.355767 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.354840 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:39.373365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.356052 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:39.402257 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.397352 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:40:39.461955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.454841 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:39.603469 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.452261 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:40:39.603757 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.452220 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:40:39.696935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.695803 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:40:39.898907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.890898 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:40:39.913924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.912367 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 21:40:39.913924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:39.912655 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:40.561939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.559181 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 21:40:40.572477 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.572423 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 21:40:40.572768 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.572754 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:40:40.572924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.572912 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:40:40.573044 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.573034 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:40:40.573149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.573139 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:40:40.596913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.590490 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:40:40.843449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.843363 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:40.851376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:40.851349 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:40:41.362521 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:41.362490 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:40:42.349508 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:40:42.349400 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:40:59.275663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.275623 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp"] Oct 30 21:40:59.283661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.283622 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.340847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.340802 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp"] Oct 30 21:40:59.446729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.446077 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.446729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.446184 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.446729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.446229 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxmc\" (UniqueName: \"kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.446729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.446495 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.456464 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.456432 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bed6793d-serving-cert\"" Oct 30 21:40:59.800907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.797431 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.800907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.797555 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxmc\" (UniqueName: \"kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.800907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.798103 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.800907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.798465 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.859392 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:40:59.859335 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:40:59.905908 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:40:59.905280 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-serving-cert: secret "maas-api-e2e-ait-bed6793d-serving-cert" not found Oct 30 21:40:59.946938 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:40:59.937425 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls podName:8a290831-08ea-4caa-a35c-f16319df723a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:00.437395377 +0000 UTC m=+1412.339259435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls") pod "maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" (UID: "8a290831-08ea-4caa-a35c-f16319df723a") : secret "maas-api-e2e-ait-bed6793d-serving-cert" not found Oct 30 21:41:00.014934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.012672 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:00.065967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.065093 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxmc\" (UniqueName: \"kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:00.504509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.504469 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:00.507173 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.507152 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\") pod \"maas-api-e2e-ait-bed6793d-657c594dfc-hptgp\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:00.507372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.507359 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:00.887955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.887924 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp"] Oct 30 21:41:00.888998 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:41:00.888963 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a290831_08ea_4caa_a35c_f16319df723a.slice/crio-6647b0e0437b502491c379d2f518ff4a5ff1f201ef7df212678a22c8f7b80f0c WatchSource:0}: Error finding container 6647b0e0437b502491c379d2f518ff4a5ff1f201ef7df212678a22c8f7b80f0c: Status 404 returned error can't find the container with id 6647b0e0437b502491c379d2f518ff4a5ff1f201ef7df212678a22c8f7b80f0c Oct 30 21:41:00.890488 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:00.890469 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:41:01.037551 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:01.037514 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" event={"ID":"8a290831-08ea-4caa-a35c-f16319df723a","Type":"ContainerStarted","Data":"6647b0e0437b502491c379d2f518ff4a5ff1f201ef7df212678a22c8f7b80f0c"} Oct 30 21:41:02.046178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:02.046085 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" event={"ID":"8a290831-08ea-4caa-a35c-f16319df723a","Type":"ContainerStarted","Data":"9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400"} Oct 30 21:41:02.046548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:02.046175 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:02.163384 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:02.163328 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" podStartSLOduration=2.38704455 podStartE2EDuration="3.163314787s" totalImagesPullingTime="776.270237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:02.160861645 +0000 UTC m=+1414.062725724" watchObservedRunningTime="2026-10-30 21:41:02.163314787 +0000 UTC m=+1414.065178852" Oct 30 21:41:03.277566 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.277525 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg"] Oct 30 21:41:03.282333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.282307 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.328853 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.328810 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f"] Oct 30 21:41:03.337007 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.335165 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351185 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351155 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351212 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351293 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351382 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351453 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351424 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351481 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351520 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351552 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351643 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4gx\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351691 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351674 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351733 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351689 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351736 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351859 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351846 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351868 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351922 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crrdq\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.351997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351972 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.351997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.351988 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.352068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.352033 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.432556 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.432515 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f"] Oct 30 21:41:03.453948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.453869 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.453948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.453949 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.453968 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crrdq\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454019 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454044 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454119 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454169 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454366 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454236 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454366 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454324 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454368 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454409 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454417 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454456 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454511 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454552 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454634 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4gx\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454723 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454687 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454723 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454711 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.454795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454728 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.454795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.454747 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.455181 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455156 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.455305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455177 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.455579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455551 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.455694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455668 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.456255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455776 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.456255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455926 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.456255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.455985 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.456255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.456241 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.458673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.458648 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.462235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.462212 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.462465 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.462442 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.462566 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.462447 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.492203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.492146 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg"] Oct 30 21:41:03.514001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.513970 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.515069 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.515037 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crrdq\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.516604 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.516572 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.519193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.519166 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4gx\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx\") pod \"e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:03.597227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.597133 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:03.653584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:03.653320 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:04.030331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.030302 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg"] Oct 30 21:41:04.031573 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:41:04.031548 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1379ca66_23f3_43b9_b98b_786ab7d9f763.slice/crio-a19665f9ecd6a5b43ce564129445fed7ec85fec759da4d4cf165c2b4c0832405 WatchSource:0}: Error finding container a19665f9ecd6a5b43ce564129445fed7ec85fec759da4d4cf165c2b4c0832405: Status 404 returned error can't find the container with id a19665f9ecd6a5b43ce564129445fed7ec85fec759da4d4cf165c2b4c0832405 Oct 30 21:41:04.034362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.034329 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.034458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.034399 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.034458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.034430 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.061446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.061410 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" event={"ID":"1379ca66-23f3-43b9-b98b-786ab7d9f763","Type":"ContainerStarted","Data":"a19665f9ecd6a5b43ce564129445fed7ec85fec759da4d4cf165c2b4c0832405"} Oct 30 21:41:04.080900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.080858 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f"] Oct 30 21:41:04.081027 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:41:04.080994 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e661bb3_f3ee_48e5_a053_06fe0a677403.slice/crio-6dd532dbbad9b85b24bc994b99a26bcba8e9c45eb3658a31c12e4f85037c2ac6 WatchSource:0}: Error finding container 6dd532dbbad9b85b24bc994b99a26bcba8e9c45eb3658a31c12e4f85037c2ac6: Status 404 returned error can't find the container with id 6dd532dbbad9b85b24bc994b99a26bcba8e9c45eb3658a31c12e4f85037c2ac6 Oct 30 21:41:04.084142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.084113 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.084238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.084177 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.084238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.084211 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:04.128505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.128477 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78"] Oct 30 21:41:04.133275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.133249 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.172275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.172243 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.172453 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.172305 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.172592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.172564 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvdj\" (UniqueName: \"kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.172763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.172740 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.185589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.185533 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d56d1199-serving-cert\"" Oct 30 21:41:04.189378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.189348 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78"] Oct 30 21:41:04.274805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.274730 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvdj\" (UniqueName: \"kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.275081 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.274929 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.275182 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.275092 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.275182 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.275156 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.275727 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:41:04.275704 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-serving-cert: secret "maas-api-e2e-migrate-d56d1199-serving-cert" not found Oct 30 21:41:04.275855 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:41:04.275771 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls podName:0e63d0eb-26c7-4d9b-89a5-172618ccf477 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:04.775749829 +0000 UTC m=+1416.677613887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls") pod "maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" (UID: "0e63d0eb-26c7-4d9b-89a5-172618ccf477") : secret "maas-api-e2e-migrate-d56d1199-serving-cert" not found Oct 30 21:41:04.277317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.277295 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.279712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.279665 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.333254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.333222 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvdj\" (UniqueName: \"kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.812164 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.812127 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:04.814737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:04.814716 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\") pod \"maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:05.056527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.056490 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:05.069948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.069912 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" event={"ID":"7e661bb3-f3ee-48e5-a053-06fe0a677403","Type":"ContainerStarted","Data":"9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a"} Oct 30 21:41:05.070083 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.069977 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" event={"ID":"7e661bb3-f3ee-48e5-a053-06fe0a677403","Type":"ContainerStarted","Data":"6dd532dbbad9b85b24bc994b99a26bcba8e9c45eb3658a31c12e4f85037c2ac6"} Oct 30 21:41:05.072664 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.072636 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" event={"ID":"1379ca66-23f3-43b9-b98b-786ab7d9f763","Type":"ContainerStarted","Data":"dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f"} Oct 30 21:41:05.425188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.425126 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podStartSLOduration=2.42510988 podStartE2EDuration="2.42510988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:05.220266038 +0000 UTC m=+1417.122130105" watchObservedRunningTime="2026-10-30 21:41:05.42510988 +0000 UTC m=+1417.326973944" Oct 30 21:41:05.427789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.427715 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podStartSLOduration=2.427699105 podStartE2EDuration="2.427699105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:05.420295776 +0000 UTC m=+1417.322159867" watchObservedRunningTime="2026-10-30 21:41:05.427699105 +0000 UTC m=+1417.329563174" Oct 30 21:41:05.535531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.535502 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78"] Oct 30 21:41:05.537564 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:41:05.537539 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e63d0eb_26c7_4d9b_89a5_172618ccf477.slice/crio-4a4b69cee654f99303385b05b36208fe664d0a7dcbf6b6fd5d9aa7017fb1dd7a WatchSource:0}: Error finding container 4a4b69cee654f99303385b05b36208fe664d0a7dcbf6b6fd5d9aa7017fb1dd7a: Status 404 returned error can't find the container with id 4a4b69cee654f99303385b05b36208fe664d0a7dcbf6b6fd5d9aa7017fb1dd7a Oct 30 21:41:05.599319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.598249 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:05.621908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.602010 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.621908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.602087 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:05.654389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.654358 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:05.655469 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.655446 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:05.655546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:05.655495 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:06.146598 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:06.145992 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" event={"ID":"0e63d0eb-26c7-4d9b-89a5-172618ccf477","Type":"ContainerStarted","Data":"4a4b69cee654f99303385b05b36208fe664d0a7dcbf6b6fd5d9aa7017fb1dd7a"} Oct 30 21:41:06.608527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:06.604519 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.608527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:06.604584 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:06.653826 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:06.653795 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:06.654012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:06.653836 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:07.154775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:07.154733 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" event={"ID":"0e63d0eb-26c7-4d9b-89a5-172618ccf477","Type":"ContainerStarted","Data":"8123292ac6a44bc561909c54901aabbc25430ddbd945266056559751a222f45e"} Oct 30 21:41:07.154991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:07.154854 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:07.660491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:07.624838 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.660491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:07.625028 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:07.660491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:07.657431 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:07.660491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:07.657498 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:08.058237 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:08.058149 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:41:08.232930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:08.232713 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" podStartSLOduration=3.307738876 podStartE2EDuration="4.232688774s" totalImagesPullingTime="924.949898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:07.337347348 +0000 UTC m=+1419.239211416" watchObservedRunningTime="2026-10-30 21:41:08.232688774 +0000 UTC m=+1420.134552844" Oct 30 21:41:08.597833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:08.597797 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:08.598197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:08.598150 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:08.654496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:08.654463 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:08.654682 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:08.654519 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:09.600365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:09.600324 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:09.630545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:09.629982 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:09.654192 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:09.654154 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:09.654496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:09.654466 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:10.603537 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:10.603493 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.604184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:10.604150 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:10.653992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:10.653956 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:10.654166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:10.654017 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:11.597933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:11.597852 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.598121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:11.597965 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 21:41:11.653736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:11.653701 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 21:41:11.654164 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:11.653753 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 21:41:12.474589 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:41:12.474552 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:41:12.744633 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:12.744532 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:12.745156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:12.745070 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:12.746002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:12.745981 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:41:12.815210 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:12.815175 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:12.815505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:12.815483 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:12.816519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:12.816495 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:41:13.171214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:13.171185 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:41:31.595408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:31.595375 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:41:36.647987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.647900 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2"] Oct 30 21:41:36.653823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.653797 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.710306 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.710271 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x8rmh\"" Oct 30 21:41:36.765591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765537 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.765799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765598 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.765799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765626 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.765799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765677 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.765799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765727 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.765799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765795 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.766116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765917 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vgk\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.766116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.765947 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.766116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.766099 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.862072 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.862031 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2"] Oct 30 21:41:36.867309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.867272 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.867565 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.867540 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.867706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.867684 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.867869 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.867851 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868015 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868027 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868508 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868491 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868653 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vgk\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868732 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868840 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868811 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868896 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.867719 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.868896 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.868314 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.869135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.869120 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.869528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.869513 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.928988 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.927265 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.938934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.935622 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.939517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.939488 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.941946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.941918 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vgk\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-bc7r2\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:36.970567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:36.970528 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:37.531386 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:37.531346 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2"] Oct 30 21:41:37.534521 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:41:37.534496 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fad0ca5_639a_474f_aa48_51184c8adbf8.slice/crio-510d0a1ccbd1b6991c323ccd42dff118201a8ff305f6035a40957ed43c61d954 WatchSource:0}: Error finding container 510d0a1ccbd1b6991c323ccd42dff118201a8ff305f6035a40957ed43c61d954: Status 404 returned error can't find the container with id 510d0a1ccbd1b6991c323ccd42dff118201a8ff305f6035a40957ed43c61d954 Oct 30 21:41:37.538180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:37.538149 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:37.538258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:37.538215 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:37.538258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:37.538244 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:37.563939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:37.563910 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" event={"ID":"5fad0ca5-639a-474f-aa48-51184c8adbf8","Type":"ContainerStarted","Data":"510d0a1ccbd1b6991c323ccd42dff118201a8ff305f6035a40957ed43c61d954"} Oct 30 21:41:38.606906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:38.589173 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" event={"ID":"5fad0ca5-639a-474f-aa48-51184c8adbf8","Type":"ContainerStarted","Data":"a6dd04de3eb79031fe011941e8ed667b0fa5b6c9054ec899b449aa224add5edf"} Oct 30 21:41:38.761636 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:38.761461 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podStartSLOduration=2.7614442930000003 podStartE2EDuration="2.761444293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:38.760379443 +0000 UTC m=+1450.662243510" watchObservedRunningTime="2026-10-30 21:41:38.761444293 +0000 UTC m=+1450.663308440" Oct 30 21:41:38.971376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:38.971348 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:38.976906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:38.972971 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:38.977131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:38.977099 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:39.972123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:39.972088 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:39.972544 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:39.972152 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:40.502775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.502736 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz"] Oct 30 21:41:40.513714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.511910 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.638912 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639006 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72csv\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639252 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639343 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639451 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639536 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639611 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639635 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.639966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.639681 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.713030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.712988 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz"] Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.740747 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.740826 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.740921 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.740955 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741012 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741034 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741077 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741235 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741650 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741778 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.742435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.741834 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72csv\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.743426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.743210 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.745100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.745056 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.745515 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.745491 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.749775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.749753 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.750114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.750084 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.805964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.805810 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.808143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.808109 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72csv\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv\") pod \"e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.837978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.837941 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:40.977915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.971479 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:40.977915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:40.971525 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:41.303979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.303946 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz"] Oct 30 21:41:41.305618 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:41:41.305576 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200158f4_1cf1_4c7e_8970_d4acbf4c1ecd.slice/crio-b0f652631d72028eb18390afc9aa7c750c2b38ed61eac676b528b669d0edc83a WatchSource:0}: Error finding container b0f652631d72028eb18390afc9aa7c750c2b38ed61eac676b528b669d0edc83a: Status 404 returned error can't find the container with id b0f652631d72028eb18390afc9aa7c750c2b38ed61eac676b528b669d0edc83a Oct 30 21:41:41.309179 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.309145 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:41.309298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.309221 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:41.309298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.309258 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:41.632233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.632193 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" event={"ID":"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd","Type":"ContainerStarted","Data":"d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee"} Oct 30 21:41:41.632447 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.632249 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" event={"ID":"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd","Type":"ContainerStarted","Data":"b0f652631d72028eb18390afc9aa7c750c2b38ed61eac676b528b669d0edc83a"} Oct 30 21:41:41.766507 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.766420 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podStartSLOduration=1.766398849 podStartE2EDuration="1.766398849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:41.759030507 +0000 UTC m=+1453.660894574" watchObservedRunningTime="2026-10-30 21:41:41.766398849 +0000 UTC m=+1453.668262915" Oct 30 21:41:41.838618 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.838572 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:41.972195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.972104 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:41.972195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:41.972182 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:42.357143 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:41:42.357050 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:41:42.854938 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:42.848855 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.854938 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:42.848934 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 21:41:42.971561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:42.971518 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:42.971764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:42.971584 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:43.172033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.171998 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:43.172245 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.172068 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:43.172245 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.172063 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:43.172245 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.172127 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:43.839143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.839100 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.839631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.839185 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 21:41:43.971363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.971308 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:43.971570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:43.971412 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:44.839253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:44.839219 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.839698 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:44.839674 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 21:41:44.971805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:44.971761 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:44.971980 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:44.971827 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:45.752117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.752029 5914 generic.go:356] "Generic (PLEG): container finished" podID="ac1ee904-e6de-4aa6-abd9-279746863133" containerID="ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d" exitCode=0 Oct 30 21:41:45.752117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.752089 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" event={"ID":"ac1ee904-e6de-4aa6-abd9-279746863133","Type":"ContainerDied","Data":"ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d"} Oct 30 21:41:45.752717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.752691 5914 scope.go:122] "RemoveContainer" containerID="ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d" Oct 30 21:41:45.754009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.753982 5914 generic.go:356] "Generic (PLEG): container finished" podID="ff1bae9c-1080-472d-b688-100c93cef310" containerID="dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed" exitCode=0 Oct 30 21:41:45.754101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.754067 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" event={"ID":"ff1bae9c-1080-472d-b688-100c93cef310","Type":"ContainerDied","Data":"dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed"} Oct 30 21:41:45.754524 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.754505 5914 scope.go:122] "RemoveContainer" containerID="dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed" Oct 30 21:41:45.755142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.754793 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:45.755142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.754986 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:45.755142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.755051 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:45.756101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.756076 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:45.756179 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.756161 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:45.756232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.756215 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:41:45.839476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.839445 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:41:45.839838 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.839502 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 21:41:45.977910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.973219 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:41:45.977910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:45.973275 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:41:46.781049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:46.780290 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" event={"ID":"ac1ee904-e6de-4aa6-abd9-279746863133","Type":"ContainerStarted","Data":"2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da"} Oct 30 21:41:46.784373 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:46.784339 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" event={"ID":"ff1bae9c-1080-472d-b688-100c93cef310","Type":"ContainerStarted","Data":"205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243"} Oct 30 21:41:46.841720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:46.841683 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:41:46.842361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:46.842330 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 21:41:47.188908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.185316 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:41:47.188908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.185452 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:47.188908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.185507 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:47.440968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.428823 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:41:47.440968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.428963 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:47.440968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.429009 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:47.865969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.864103 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 21:41:47.865969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:47.864159 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 21:41:48.067338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.065555 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:41:48.067338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.066008 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:41:48.067338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.066078 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:41:48.193417 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.193383 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:48.193703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.193677 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:48.433948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.428758 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:48.433948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.428823 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:48.531414 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.526707 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:48.531414 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.528345 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:48.554640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.549390 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:41:48.847869 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.846263 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:48.847869 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.847838 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:48.848801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:48.848777 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:41:49.204911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:49.181322 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:49.235906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:49.212974 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:49.427349 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:49.427308 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:49.427567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:49.427392 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:50.199907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:50.186377 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:50.199907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:50.186443 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:50.426905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:50.426680 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:50.426905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:50.426744 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:51.177038 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:51.176993 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:51.177259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:51.177072 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:51.427526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:51.427413 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:51.427526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:51.427502 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:52.175904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:52.174284 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:41:52.175904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:52.174358 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:41:52.426966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:52.426278 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 21:41:52.426966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:52.426348 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 21:41:53.176957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:53.176927 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:41:53.177337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:53.177207 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:41:53.177981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:53.177962 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:41:53.432415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:53.432332 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:41:53.432864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:53.432835 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:41:53.433796 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:41:53.433775 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:42:12.347274 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:12.347226 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:42:16.359211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:16.359180 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk"] Oct 30 21:42:16.476213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:16.476176 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n"] Oct 30 21:42:21.364066 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.363957 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" podUID="ecb379de-f7c9-45fe-84e5-3211b6499794" containerName="payload-processing" containerID="cri-o://48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61" gracePeriod=25 Oct 30 21:42:21.477177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.477128 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" podUID="a7c44cc1-e025-4ab0-8691-bf42d4c846c8" containerName="payload-pre-processing" containerID="cri-o://1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3" gracePeriod=25 Oct 30 21:42:21.657661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.657633 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:42:21.772990 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.772965 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:42:21.773248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.773227 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd\" (UniqueName: \"kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd\") pod \"ecb379de-f7c9-45fe-84e5-3211b6499794\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " Oct 30 21:42:21.773335 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.773270 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume\") pod \"ecb379de-f7c9-45fe-84e5-3211b6499794\" (UID: \"ecb379de-f7c9-45fe-84e5-3211b6499794\") " Oct 30 21:42:21.774018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.773987 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume" pod "ecb379de-f7c9-45fe-84e5-3211b6499794" (UID: "ecb379de-f7c9-45fe-84e5-3211b6499794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:21.774505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.774479 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb379de-f7c9-45fe-84e5-3211b6499794-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:21.780720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.780695 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd" pod "ecb379de-f7c9-45fe-84e5-3211b6499794" (UID: "ecb379de-f7c9-45fe-84e5-3211b6499794"). InnerVolumeSpecName "kube-api-access-4jqcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:21.875251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.875224 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume\") pod \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " Oct 30 21:42:21.875401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.875360 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4\" (UniqueName: \"kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4\") pod \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\" (UID: \"a7c44cc1-e025-4ab0-8691-bf42d4c846c8\") " Oct 30 21:42:21.875675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.875646 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume" pod "a7c44cc1-e025-4ab0-8691-bf42d4c846c8" (UID: "a7c44cc1-e025-4ab0-8691-bf42d4c846c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:21.876003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.875986 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:21.876061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.876006 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jqcd\" (UniqueName: \"kubernetes.io/projected/ecb379de-f7c9-45fe-84e5-3211b6499794-kube-api-access-4jqcd\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:21.877627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.877597 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4" pod "a7c44cc1-e025-4ab0-8691-bf42d4c846c8" (UID: "a7c44cc1-e025-4ab0-8691-bf42d4c846c8"). InnerVolumeSpecName "kube-api-access-2q4q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:21.977440 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:21.977398 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4q4\" (UniqueName: \"kubernetes.io/projected/a7c44cc1-e025-4ab0-8691-bf42d4c846c8-kube-api-access-2q4q4\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:22.217030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.216923 5914 generic.go:356] "Generic (PLEG): container finished" podID="a7c44cc1-e025-4ab0-8691-bf42d4c846c8" containerID="1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3" exitCode=0 Oct 30 21:42:22.217211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.217035 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" Oct 30 21:42:22.218703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.218679 5914 generic.go:356] "Generic (PLEG): container finished" podID="ecb379de-f7c9-45fe-84e5-3211b6499794" containerID="48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61" exitCode=0 Oct 30 21:42:22.218827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.218781 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" Oct 30 21:42:22.221358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.221333 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" event={"ID":"a7c44cc1-e025-4ab0-8691-bf42d4c846c8","Type":"ContainerDied","Data":"1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3"} Oct 30 21:42:22.221463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.221393 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n" event={"ID":"a7c44cc1-e025-4ab0-8691-bf42d4c846c8","Type":"ContainerDied","Data":"55595e93cd05263992d595aa9d5a4eb2b202304d505f10550e57b392e4ff7b8d"} Oct 30 21:42:22.221463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.221418 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" event={"ID":"ecb379de-f7c9-45fe-84e5-3211b6499794","Type":"ContainerDied","Data":"48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61"} Oct 30 21:42:22.221463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.221443 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk" event={"ID":"ecb379de-f7c9-45fe-84e5-3211b6499794","Type":"ContainerDied","Data":"ffe5800676f086078e3b9ad76a02ed17b68d89644a3c8f896191d8180fd8e5ab"} Oct 30 21:42:22.221642 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.221472 5914 scope.go:122] "RemoveContainer" containerID="1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3" Oct 30 21:42:22.234919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.234523 5914 scope.go:122] "RemoveContainer" containerID="1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3" Oct 30 21:42:22.234998 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:22.234947 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3\": container with ID starting with 1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3 not found: ID does not exist" containerID="1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3" Oct 30 21:42:22.235040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.235012 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3"} err="failed to get container status \"1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3\": rpc error: code = NotFound desc = could not find container \"1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3\": container with ID starting with 1d6dc9983d3e3ecc5caede4d852a57093cbc9db62133120c51df39cd227e54f3 not found: ID does not exist" Oct 30 21:42:22.235040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.235033 5914 scope.go:122] "RemoveContainer" containerID="48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61" Oct 30 21:42:22.244780 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.244760 5914 scope.go:122] "RemoveContainer" containerID="48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61" Oct 30 21:42:22.245088 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:22.245070 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61\": container with ID starting with 48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61 not found: ID does not exist" containerID="48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61" Oct 30 21:42:22.245144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.245095 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61"} err="failed to get container status \"48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61\": rpc error: code = NotFound desc = could not find container \"48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61\": container with ID starting with 48d591c5756058de72f3bfe64edf6a4ae32620a73ac74b937a41b164495a3e61 not found: ID does not exist" Oct 30 21:42:22.383286 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.383255 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n"] Oct 30 21:42:22.388339 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.388313 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d56d1199-68d587b47f-h5c8n"] Oct 30 21:42:22.493224 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.493130 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk"] Oct 30 21:42:22.497057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:22.497024 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d56d1199-5b88959bcf-5khrk"] Oct 30 21:42:23.002988 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:23.002954 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2"] Oct 30 21:42:24.219752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:24.219722 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c44cc1-e025-4ab0-8691-bf42d4c846c8" path="/var/lib/kubelet/pods/a7c44cc1-e025-4ab0-8691-bf42d4c846c8/volumes" Oct 30 21:42:24.220265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:24.220249 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb379de-f7c9-45fe-84e5-3211b6499794" path="/var/lib/kubelet/pods/ecb379de-f7c9-45fe-84e5-3211b6499794/volumes" Oct 30 21:42:28.004185 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.004145 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" podUID="8234fd05-22f4-4591-81f7-c0281ffa1e72" containerName="payload-processing" containerID="cri-o://2bb44f9b3145b723cbfd3fed8c2f8f32b3d48a3cf5b0115777bbf99f7d20e96f" gracePeriod=25 Oct 30 21:42:28.324943 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.324807 5914 generic.go:356] "Generic (PLEG): container finished" podID="8234fd05-22f4-4591-81f7-c0281ffa1e72" containerID="2bb44f9b3145b723cbfd3fed8c2f8f32b3d48a3cf5b0115777bbf99f7d20e96f" exitCode=0 Oct 30 21:42:28.325073 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.324934 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" event={"ID":"8234fd05-22f4-4591-81f7-c0281ffa1e72","Type":"ContainerDied","Data":"2bb44f9b3145b723cbfd3fed8c2f8f32b3d48a3cf5b0115777bbf99f7d20e96f"} Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.381018 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx"] Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.382154 5914 cpu_manager.go:414] "containerMap: removing container" podUID="a7c44cc1-e025-4ab0-8691-bf42d4c846c8" containerName="payload-pre-processing" Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.382176 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c44cc1-e025-4ab0-8691-bf42d4c846c8" containerName="payload-pre-processing" Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.382235 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ecb379de-f7c9-45fe-84e5-3211b6499794" containerName="payload-processing" Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.382244 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb379de-f7c9-45fe-84e5-3211b6499794" containerName="payload-processing" Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.382405 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb379de-f7c9-45fe-84e5-3211b6499794" containerName="payload-processing" Oct 30 21:42:28.391054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.382426 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c44cc1-e025-4ab0-8691-bf42d4c846c8" containerName="payload-pre-processing" Oct 30 21:42:28.799279 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.799244 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx"] Oct 30 21:42:28.799867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.799831 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.959393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.957676 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-3b21b4-openshift-default-dockercfg-wlwtn\"" Oct 30 21:42:28.973285 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973251 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.973467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973422 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.973591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973573 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.973706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973636 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84jz4\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.973706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973699 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.973829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973786 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.973975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.973910 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.974325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.974302 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:28.974413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:28.974371 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.009133 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.009098 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:42:29.077585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.077558 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:42:29.077760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.077600 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:42:29.123686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.123653 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume\") pod \"8234fd05-22f4-4591-81f7-c0281ffa1e72\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " Oct 30 21:42:29.123868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.123794 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd\" (UniqueName: \"kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd\") pod \"8234fd05-22f4-4591-81f7-c0281ffa1e72\" (UID: \"8234fd05-22f4-4591-81f7-c0281ffa1e72\") " Oct 30 21:42:29.124088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124062 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume" pod "8234fd05-22f4-4591-81f7-c0281ffa1e72" (UID: "8234fd05-22f4-4591-81f7-c0281ffa1e72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:29.124200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124181 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124280 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124266 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124316 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124288 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124477 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124459 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124550 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124645 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124562 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124743 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124687 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124743 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124717 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84jz4\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124750 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.124861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124815 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.125033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124857 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.125033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.124932 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.125192 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.125160 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.125267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.125206 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8234fd05-22f4-4591-81f7-c0281ffa1e72-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:29.125362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.125342 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.126426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.126402 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd" pod "8234fd05-22f4-4591-81f7-c0281ffa1e72" (UID: "8234fd05-22f4-4591-81f7-c0281ffa1e72"). InnerVolumeSpecName "kube-api-access-hxrjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:29.126785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.126765 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.127489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.127470 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.182484 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.182440 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.182484 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.182474 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84jz4\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4\") pod \"e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.226358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.226328 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrjd\" (UniqueName: \"kubernetes.io/projected/8234fd05-22f4-4591-81f7-c0281ffa1e72-kube-api-access-hxrjd\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:29.333505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.333410 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" Oct 30 21:42:29.333505 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.333449 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2" event={"ID":"8234fd05-22f4-4591-81f7-c0281ffa1e72","Type":"ContainerDied","Data":"fea1cd9561484a3301f1e097f72cefb874c1dc8ad26767e5b3e7839a213baea9"} Oct 30 21:42:29.333703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.333525 5914 scope.go:122] "RemoveContainer" containerID="2bb44f9b3145b723cbfd3fed8c2f8f32b3d48a3cf5b0115777bbf99f7d20e96f" Oct 30 21:42:29.351135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.351111 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:29.500465 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.500432 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2"] Oct 30 21:42:29.504128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.504101 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-385396af-64b8655bd5-tckp2"] Oct 30 21:42:29.788188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.788160 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx"] Oct 30 21:42:29.788350 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:42:29.788325 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bbf7f4_758d_497d_9bfe_a4273c871f63.slice/crio-eebc189db4ac603ff4822fb031c68426abcfd4eed638949bd7d3062de1202c1d WatchSource:0}: Error finding container eebc189db4ac603ff4822fb031c68426abcfd4eed638949bd7d3062de1202c1d: Status 404 returned error can't find the container with id eebc189db4ac603ff4822fb031c68426abcfd4eed638949bd7d3062de1202c1d Oct 30 21:42:29.791109 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.791080 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:42:29.791195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.791139 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:42:29.791195 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:29.791171 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:42:30.222698 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:30.222664 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8234fd05-22f4-4591-81f7-c0281ffa1e72" path="/var/lib/kubelet/pods/8234fd05-22f4-4591-81f7-c0281ffa1e72/volumes" Oct 30 21:42:30.341529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:30.341482 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" event={"ID":"81bbf7f4-758d-497d-9bfe-a4273c871f63","Type":"ContainerStarted","Data":"04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b"} Oct 30 21:42:30.341690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:30.341558 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" event={"ID":"81bbf7f4-758d-497d-9bfe-a4273c871f63","Type":"ContainerStarted","Data":"eebc189db4ac603ff4822fb031c68426abcfd4eed638949bd7d3062de1202c1d"} Oct 30 21:42:30.351872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:30.351846 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:30.357518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:30.357497 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:30.462214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:30.462162 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podStartSLOduration=2.462146733 podStartE2EDuration="2.462146733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:30.460222784 +0000 UTC m=+1502.362086851" watchObservedRunningTime="2026-10-30 21:42:30.462146733 +0000 UTC m=+1502.364010798" Oct 30 21:42:31.349852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:31.349820 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:31.350977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:31.350959 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:42:42.347042 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:42.347003 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:42:43.853027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:43.852987 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:42:46.745121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:46.745087 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp"] Oct 30 21:42:46.745660 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:46.745300 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" podUID="8a290831-08ea-4caa-a35c-f16319df723a" containerName="maas-api" containerID="cri-o://9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400" gracePeriod=30 Oct 30 21:42:47.300970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.299184 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:42:47.397905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.396725 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc\" (UniqueName: \"kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc\") pod \"8a290831-08ea-4caa-a35c-f16319df723a\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " Oct 30 21:42:47.397905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.397011 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs\") pod \"8a290831-08ea-4caa-a35c-f16319df723a\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " Oct 30 21:42:47.397905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.397147 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca\") pod \"8a290831-08ea-4caa-a35c-f16319df723a\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " Oct 30 21:42:47.397905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.397358 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\") pod \"8a290831-08ea-4caa-a35c-f16319df723a\" (UID: \"8a290831-08ea-4caa-a35c-f16319df723a\") " Oct 30 21:42:47.401911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.399581 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca" pod "8a290831-08ea-4caa-a35c-f16319df723a" (UID: "8a290831-08ea-4caa-a35c-f16319df723a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:47.418395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.418359 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls" pod "8a290831-08ea-4caa-a35c-f16319df723a" (UID: "8a290831-08ea-4caa-a35c-f16319df723a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:47.419129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.419104 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs" pod "8a290831-08ea-4caa-a35c-f16319df723a" (UID: "8a290831-08ea-4caa-a35c-f16319df723a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:47.431903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.428685 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc" pod "8a290831-08ea-4caa-a35c-f16319df723a" (UID: "8a290831-08ea-4caa-a35c-f16319df723a"). InnerVolumeSpecName "kube-api-access-fbxmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:47.467624 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.467577 5914 generic.go:356] "Generic (PLEG): container finished" podID="8a290831-08ea-4caa-a35c-f16319df723a" containerID="9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400" exitCode=0 Oct 30 21:42:47.467786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.467680 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" Oct 30 21:42:47.467786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.467671 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" event={"ID":"8a290831-08ea-4caa-a35c-f16319df723a","Type":"ContainerDied","Data":"9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400"} Oct 30 21:42:47.467939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.467826 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp" event={"ID":"8a290831-08ea-4caa-a35c-f16319df723a","Type":"ContainerDied","Data":"6647b0e0437b502491c379d2f518ff4a5ff1f201ef7df212678a22c8f7b80f0c"} Oct 30 21:42:47.467939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.467870 5914 scope.go:122] "RemoveContainer" containerID="9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400" Oct 30 21:42:47.480061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.480033 5914 scope.go:122] "RemoveContainer" containerID="9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400" Oct 30 21:42:47.480360 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:47.480341 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400\": container with ID starting with 9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400 not found: ID does not exist" containerID="9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400" Oct 30 21:42:47.480442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.480372 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400"} err="failed to get container status \"9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400\": rpc error: code = NotFound desc = could not find container \"9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400\": container with ID starting with 9c7279d49e3ebcd361de8da20f82398b5dd1f9065f48762c73f2e7b9d25f5400 not found: ID does not exist" Oct 30 21:42:47.499492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.499455 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:47.499492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.499492 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a290831-08ea-4caa-a35c-f16319df723a-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:47.499492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.499502 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a290831-08ea-4caa-a35c-f16319df723a-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:47.499694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.499514 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxmc\" (UniqueName: \"kubernetes.io/projected/8a290831-08ea-4caa-a35c-f16319df723a-kube-api-access-fbxmc\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:47.639603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.639570 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp"] Oct 30 21:42:47.646489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.646458 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bed6793d-657c594dfc-hptgp"] Oct 30 21:42:47.967372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.967334 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78"] Oct 30 21:42:47.974768 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:47.974719 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" podUID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" containerName="maas-api" containerID="cri-o://8123292ac6a44bc561909c54901aabbc25430ddbd945266056559751a222f45e" gracePeriod=30 Oct 30 21:42:48.311822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.310633 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a290831-08ea-4caa-a35c-f16319df723a" path="/var/lib/kubelet/pods/8a290831-08ea-4caa-a35c-f16319df723a/volumes" Oct 30 21:42:48.630151 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.628092 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:42:48.634542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.632358 5914 generic.go:356] "Generic (PLEG): container finished" podID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" containerID="8123292ac6a44bc561909c54901aabbc25430ddbd945266056559751a222f45e" exitCode=0 Oct 30 21:42:48.634542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.632497 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" event={"ID":"0e63d0eb-26c7-4d9b-89a5-172618ccf477","Type":"ContainerDied","Data":"8123292ac6a44bc561909c54901aabbc25430ddbd945266056559751a222f45e"} Oct 30 21:42:48.634542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.632601 5914 scope.go:122] "RemoveContainer" containerID="8123292ac6a44bc561909c54901aabbc25430ddbd945266056559751a222f45e" Oct 30 21:42:48.756987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.756948 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs\") pod \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " Oct 30 21:42:48.757158 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.757122 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\") pod \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " Oct 30 21:42:48.757222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.757210 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca\") pod \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " Oct 30 21:42:48.757292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.757280 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj\" (UniqueName: \"kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj\") pod \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\" (UID: \"0e63d0eb-26c7-4d9b-89a5-172618ccf477\") " Oct 30 21:42:48.757586 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.757560 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca" pod "0e63d0eb-26c7-4d9b-89a5-172618ccf477" (UID: "0e63d0eb-26c7-4d9b-89a5-172618ccf477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:48.757987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.757968 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e63d0eb-26c7-4d9b-89a5-172618ccf477-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:48.759338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.759304 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs" pod "0e63d0eb-26c7-4d9b-89a5-172618ccf477" (UID: "0e63d0eb-26c7-4d9b-89a5-172618ccf477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:48.759473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.759458 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj" pod "0e63d0eb-26c7-4d9b-89a5-172618ccf477" (UID: "0e63d0eb-26c7-4d9b-89a5-172618ccf477"). InnerVolumeSpecName "kube-api-access-mfvdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:48.759517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.759506 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls" pod "0e63d0eb-26c7-4d9b-89a5-172618ccf477" (UID: "0e63d0eb-26c7-4d9b-89a5-172618ccf477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:42:48.862138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.862009 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:48.862940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.862143 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e63d0eb-26c7-4d9b-89a5-172618ccf477-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:48.862940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:48.862901 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfvdj\" (UniqueName: \"kubernetes.io/projected/0e63d0eb-26c7-4d9b-89a5-172618ccf477-kube-api-access-mfvdj\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:49.121637 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.121548 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc"] Oct 30 21:42:49.122321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.121831 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" containerID="cri-o://205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243" gracePeriod=30 Oct 30 21:42:49.125584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.125554 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz"] Oct 30 21:42:49.125801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.125776 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" containerID="cri-o://d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee" gracePeriod=30 Oct 30 21:42:49.761924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.761856 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" event={"ID":"0e63d0eb-26c7-4d9b-89a5-172618ccf477","Type":"ContainerDied","Data":"4a4b69cee654f99303385b05b36208fe664d0a7dcbf6b6fd5d9aa7017fb1dd7a"} Oct 30 21:42:49.762351 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.762329 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" Oct 30 21:42:49.961081 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.960995 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78"] Oct 30 21:42:49.982071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:49.981589 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78"] Oct 30 21:42:50.297598 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:50.292745 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" path="/var/lib/kubelet/pods/0e63d0eb-26c7-4d9b-89a5-172618ccf477/volumes" Oct 30 21:42:50.499400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:50.499366 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn"] Oct 30 21:42:50.499649 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:50.499571 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" containerID="cri-o://2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da" gracePeriod=30 Oct 30 21:42:51.177495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.177445 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d56d1199-74498ccc5f-9nc78" podUID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": context deadline exceeded" Oct 30 21:42:51.692018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.691001 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95"] Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.693546 5914 cpu_manager.go:414] "containerMap: removing container" podUID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" containerName="maas-api" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.693574 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" containerName="maas-api" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.693615 5914 cpu_manager.go:414] "containerMap: removing container" podUID="8a290831-08ea-4caa-a35c-f16319df723a" containerName="maas-api" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.693622 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a290831-08ea-4caa-a35c-f16319df723a" containerName="maas-api" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.693723 5914 cpu_manager.go:414] "containerMap: removing container" podUID="8234fd05-22f4-4591-81f7-c0281ffa1e72" containerName="payload-processing" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.693730 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8234fd05-22f4-4591-81f7-c0281ffa1e72" containerName="payload-processing" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.694087 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e63d0eb-26c7-4d9b-89a5-172618ccf477" containerName="maas-api" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.694113 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a290831-08ea-4caa-a35c-f16319df723a" containerName="maas-api" Oct 30 21:42:51.694647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.694132 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8234fd05-22f4-4591-81f7-c0281ffa1e72" containerName="payload-processing" Oct 30 21:42:51.725938 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.724204 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.796936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.778618 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9173600a-openshift-default-dockercfg-25q9c\"" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939423 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939568 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939636 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939665 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939729 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghsx\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939817 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.939910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.939858 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.940381 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.940029 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:51.940381 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:51.940154 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.024206 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.024116 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95"] Oct 30 21:42:52.049428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.049391 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.049833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.049798 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.062787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.062739 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.063148 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.063127 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.063319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.063305 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.063438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.063414 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.063706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.063640 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghsx\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.063925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.063906 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.064129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.064109 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.066003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.058506 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.066502 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.066481 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.066912 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.066869 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.067321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.067300 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.068652 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.068627 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.112519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.112467 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.138841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.138803 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.326110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.326030 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghsx\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.387103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.353421 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token\") pod \"e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:52.569172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:52.569132 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:53.025125 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.025098 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95"] Oct 30 21:42:53.026191 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:42:53.026165 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e095191_51ca_4537_8220_6bc220e6abce.slice/crio-d3b6ce8057e4f44068c4347f83bc29fdf9b73570c8cb2549d8f3e0300e971866 WatchSource:0}: Error finding container d3b6ce8057e4f44068c4347f83bc29fdf9b73570c8cb2549d8f3e0300e971866: Status 404 returned error can't find the container with id d3b6ce8057e4f44068c4347f83bc29fdf9b73570c8cb2549d8f3e0300e971866 Oct 30 21:42:53.028870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.028839 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:42:53.028978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.028946 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:42:53.029016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.028985 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:42:53.180260 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.180216 5914 patch_prober.go:29] interesting pod/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:42:53.180416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.180285 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:53.435557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.435515 5914 patch_prober.go:29] interesting pod/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:42:53.435727 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.435575 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:53.951073 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.951036 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" event={"ID":"1e095191-51ca-4537-8220-6bc220e6abce","Type":"ContainerStarted","Data":"4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993"} Oct 30 21:42:53.951274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:53.951091 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" event={"ID":"1e095191-51ca-4537-8220-6bc220e6abce","Type":"ContainerStarted","Data":"d3b6ce8057e4f44068c4347f83bc29fdf9b73570c8cb2549d8f3e0300e971866"} Oct 30 21:42:54.180024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.179976 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podStartSLOduration=3.179959833 podStartE2EDuration="3.179959833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:54.129927342 +0000 UTC m=+1526.031791408" watchObservedRunningTime="2026-10-30 21:42:54.179959833 +0000 UTC m=+1526.081823897" Oct 30 21:42:54.556097 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.556071 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:42:54.570315 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.570283 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:54.575568 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.575540 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:54.599702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.599677 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.599844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.599768 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.599844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.599819 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.599852 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.599909 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600016 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600063 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600107 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:54.600264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600213 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.600352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600314 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.600352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600342 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600391 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token\") pod \"ff1bae9c-1080-472d-b688-100c93cef310\" (UID: \"ff1bae9c-1080-472d-b688-100c93cef310\") " Oct 30 21:42:54.600551 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600531 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.600955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.600898 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.601207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.601190 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff1bae9c-1080-472d-b688-100c93cef310-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.601207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.601207 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.601353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.601216 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.601353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.601225 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.601353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.601235 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.602655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.602631 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:54.602735 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.602673 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.602913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.602863 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:54.603326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.603303 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb" pod "ff1bae9c-1080-472d-b688-100c93cef310" (UID: "ff1bae9c-1080-472d-b688-100c93cef310"). InnerVolumeSpecName "kube-api-access-fncwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:54.616515 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.616491 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:42:54.702014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.701981 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702234 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702211 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702273 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702319 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702363 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702393 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702560 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702542 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702618 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702606 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token\") pod \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\" (UID: \"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd\") " Oct 30 21:42:54.702967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.702935 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.703182 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.703159 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.703816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.703792 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:54.704012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.703989 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.704113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704097 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.704178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704132 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fncwb\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-kube-api-access-fncwb\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.704178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704149 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff1bae9c-1080-472d-b688-100c93cef310-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.704178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704162 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.704341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704192 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff1bae9c-1080-472d-b688-100c93cef310-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.704341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704206 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff1bae9c-1080-472d-b688-100c93cef310-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.704566 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.704443 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.705564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.705528 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "kube-api-access-72csv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:54.705839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.705695 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:54.706645 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.706625 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:54.707230 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.707199 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo" pod "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" (UID: "200158f4-1cf1-4c7e-8970-d4acbf4c1ecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806273 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806301 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806311 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806322 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806332 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72csv\" (UniqueName: \"kubernetes.io/projected/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-kube-api-access-72csv\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806342 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.806350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.806352 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:54.959068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.959033 5914 generic.go:356] "Generic (PLEG): container finished" podID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerID="d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee" exitCode=0 Oct 30 21:42:54.959259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.959080 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" Oct 30 21:42:54.959259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.959104 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" event={"ID":"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd","Type":"ContainerDied","Data":"d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee"} Oct 30 21:42:54.959259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.959162 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz" event={"ID":"200158f4-1cf1-4c7e-8970-d4acbf4c1ecd","Type":"ContainerDied","Data":"b0f652631d72028eb18390afc9aa7c750c2b38ed61eac676b528b669d0edc83a"} Oct 30 21:42:54.959259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.959190 5914 scope.go:122] "RemoveContainer" containerID="d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee" Oct 30 21:42:54.961265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.961240 5914 generic.go:356] "Generic (PLEG): container finished" podID="ff1bae9c-1080-472d-b688-100c93cef310" containerID="205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243" exitCode=0 Oct 30 21:42:54.961406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.961290 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" Oct 30 21:42:54.961406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.961318 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" event={"ID":"ff1bae9c-1080-472d-b688-100c93cef310","Type":"ContainerDied","Data":"205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243"} Oct 30 21:42:54.961406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.961362 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc" event={"ID":"ff1bae9c-1080-472d-b688-100c93cef310","Type":"ContainerDied","Data":"15247a86fde505d3a19ab3192100f82353e26a2ac56c2c6bf68c38de48192970"} Oct 30 21:42:54.961693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.961672 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:54.962836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.962805 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:42:54.973456 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.973436 5914 scope.go:122] "RemoveContainer" containerID="d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee" Oct 30 21:42:54.973937 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:54.973848 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee\": container with ID starting with d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee not found: ID does not exist" containerID="d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee" Oct 30 21:42:54.973937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.973925 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee"} err="failed to get container status \"d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee\": rpc error: code = NotFound desc = could not find container \"d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee\": container with ID starting with d60ce055154faa7af252c7cb1a0ae50b9dd27218f0cc574728bcadfb172200ee not found: ID does not exist" Oct 30 21:42:54.974082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.973949 5914 scope.go:122] "RemoveContainer" containerID="205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243" Oct 30 21:42:54.984466 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.984442 5914 scope.go:122] "RemoveContainer" containerID="dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed" Oct 30 21:42:54.995059 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.995041 5914 scope.go:122] "RemoveContainer" containerID="205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243" Oct 30 21:42:54.995376 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:54.995353 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243\": container with ID starting with 205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243 not found: ID does not exist" containerID="205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243" Oct 30 21:42:54.995473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.995388 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243"} err="failed to get container status \"205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243\": rpc error: code = NotFound desc = could not find container \"205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243\": container with ID starting with 205ad2c3f9a5a37429371d296dedd081e3b597fb92cff4e2007afe4db97d1243 not found: ID does not exist" Oct 30 21:42:54.995473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.995412 5914 scope.go:122] "RemoveContainer" containerID="dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed" Oct 30 21:42:54.995713 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:54.995694 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed\": container with ID starting with dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed not found: ID does not exist" containerID="dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed" Oct 30 21:42:54.995784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:54.995720 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed"} err="failed to get container status \"dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed\": rpc error: code = NotFound desc = could not find container \"dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed\": container with ID starting with dbc7401b820057e6610b300199f0ca4e9a28537a36f09e0a5a70e6a1522ce4ed not found: ID does not exist" Oct 30 21:42:55.373486 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.373447 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc"] Oct 30 21:42:55.383571 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.383535 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-z44vc"] Oct 30 21:42:55.502730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.502636 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz"] Oct 30 21:42:55.514059 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.514034 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bed6793d-openshift-default-649f995d97-tmgmz"] Oct 30 21:42:55.779925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.779892 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:42:55.821513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.821475 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.821681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.821524 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.821681 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.821599 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.821867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.821847 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.821967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.821936 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.822012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.821977 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.822132 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822113 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.822200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822119 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.822200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822149 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.822310 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822202 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.822310 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822242 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert\") pod \"ac1ee904-e6de-4aa6-abd9-279746863133\" (UID: \"ac1ee904-e6de-4aa6-abd9-279746863133\") " Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822470 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.822731 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.823304 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.823329 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.823345 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.823358 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1ee904-e6de-4aa6-abd9-279746863133-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.823450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.823423 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.824502 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.824475 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:42:55.824624 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.824603 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:55.824801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.824774 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:42:55.824801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.824791 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms" pod "ac1ee904-e6de-4aa6-abd9-279746863133" (UID: "ac1ee904-e6de-4aa6-abd9-279746863133"). InnerVolumeSpecName "kube-api-access-94gms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:42:55.925506 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.925443 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94gms\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-kube-api-access-94gms\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.925506 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.925495 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.925506 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.925508 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1ee904-e6de-4aa6-abd9-279746863133-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.925506 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.925517 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1ee904-e6de-4aa6-abd9-279746863133-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.925850 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.925529 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1ee904-e6de-4aa6-abd9-279746863133-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:42:55.970197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.970162 5914 generic.go:356] "Generic (PLEG): container finished" podID="ac1ee904-e6de-4aa6-abd9-279746863133" containerID="2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da" exitCode=0 Oct 30 21:42:55.970376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.970216 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" Oct 30 21:42:55.970376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.970243 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" event={"ID":"ac1ee904-e6de-4aa6-abd9-279746863133","Type":"ContainerDied","Data":"2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da"} Oct 30 21:42:55.970376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.970323 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn" event={"ID":"ac1ee904-e6de-4aa6-abd9-279746863133","Type":"ContainerDied","Data":"9a6a36ea22961a1cfb82ffe793c89dd6ed76416c305a1fc71f901f2671243173"} Oct 30 21:42:55.970376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.970348 5914 scope.go:122] "RemoveContainer" containerID="2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da" Oct 30 21:42:55.981758 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.981735 5914 scope.go:122] "RemoveContainer" containerID="ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d" Oct 30 21:42:55.991400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.991383 5914 scope.go:122] "RemoveContainer" containerID="2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da" Oct 30 21:42:55.991695 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:55.991672 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da\": container with ID starting with 2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da not found: ID does not exist" containerID="2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da" Oct 30 21:42:55.991734 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.991704 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da"} err="failed to get container status \"2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da\": rpc error: code = NotFound desc = could not find container \"2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da\": container with ID starting with 2daeba191d465ba4c124b4a9afe632a0c7dfd76a9cf14e9cee39672058be78da not found: ID does not exist" Oct 30 21:42:55.991734 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.991722 5914 scope.go:122] "RemoveContainer" containerID="ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d" Oct 30 21:42:55.992053 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:42:55.992037 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d\": container with ID starting with ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d not found: ID does not exist" containerID="ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d" Oct 30 21:42:55.992100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:55.992058 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d"} err="failed to get container status \"ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d\": rpc error: code = NotFound desc = could not find container \"ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d\": container with ID starting with ac57f821f7b5640e20836f4ec7ca7c5a97e77e783d2f217ba9d9c141a41b942d not found: ID does not exist" Oct 30 21:42:56.141210 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:56.141177 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn"] Oct 30 21:42:56.144438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:56.144410 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d56d1199-gw-openshift-default-b79f87457-8rffn"] Oct 30 21:42:56.219571 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:56.219536 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" path="/var/lib/kubelet/pods/200158f4-1cf1-4c7e-8970-d4acbf4c1ecd/volumes" Oct 30 21:42:56.220120 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:56.220104 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" path="/var/lib/kubelet/pods/ac1ee904-e6de-4aa6-abd9-279746863133/volumes" Oct 30 21:42:56.220572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:56.220555 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1bae9c-1080-472d-b688-100c93cef310" path="/var/lib/kubelet/pods/ff1bae9c-1080-472d-b688-100c93cef310/volumes" Oct 30 21:42:59.965184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.965146 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs"] Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.970949 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.970995 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971037 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971046 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971093 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971102 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971113 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971121 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971163 5914 cpu_manager.go:414] "containerMap: removing container" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971171 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971448 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971475 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1ee904-e6de-4aa6-abd9-279746863133" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971508 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971519 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1bae9c-1080-472d-b688-100c93cef310" containerName="istio-proxy" Oct 30 21:42:59.975977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.971529 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="200158f4-1cf1-4c7e-8970-d4acbf4c1ecd" containerName="istio-proxy" Oct 30 21:42:59.980902 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:42:59.977686 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.044586 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.043226 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs"] Oct 30 21:43:00.082934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.080395 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9173600a\"" Oct 30 21:43:00.100902 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.095292 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72"] Oct 30 21:43:00.116914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.115380 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.163606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.162996 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqgnm\" (UniqueName: \"kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm\") pod \"payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.163606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.163114 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.181929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.173846 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9173600a-dockercfg-gjxn9\"" Oct 30 21:43:00.205907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.203216 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72"] Oct 30 21:43:00.267864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.266035 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqgnm\" (UniqueName: \"kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm\") pod \"payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.267864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.266229 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcd7p\" (UniqueName: \"kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p\") pod \"payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.267864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.266319 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.267864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.266392 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume\") pod \"payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.267864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.267805 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.370925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.369932 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcd7p\" (UniqueName: \"kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p\") pod \"payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.370925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.370071 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume\") pod \"payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.398378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.371482 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume\") pod \"payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.454390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.410568 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqgnm\" (UniqueName: \"kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm\") pod \"payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.454390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.412672 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:00.572482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.492381 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcd7p\" (UniqueName: \"kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p\") pod \"payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.572482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.539527 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:00.572482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.570651 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8"] Oct 30 21:43:00.685775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.673489 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8"] Oct 30 21:43:00.685775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.673640 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.769224 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.768297 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1915737c-openshift-default-dockercfg-zjl7p\"" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.816740 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.816862 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833182 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqvm\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833335 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833440 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833477 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833518 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833555 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.834209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.833658 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.950620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.939952 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.965224 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.957178 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqvm\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.966165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.966130 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.966721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.966660 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.967089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.967068 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.967407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.967335 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.967693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.967675 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.968358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.968018 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.969322 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.969281 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.969619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.951270 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:00.972676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:00.972645 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.019470 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.019436 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.020137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.020114 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.077995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.020815 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.078231 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.024232 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.078341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.024436 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.121694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.101026 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.226038 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.225966 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqvm\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm\") pod \"e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.232918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.231815 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:01.455391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.455357 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs"] Oct 30 21:43:01.455986 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:43:01.455954 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bef2b3b_edb4_4272_823f_e7c0e035f082.slice/crio-f03a7251428414b5ea2a6bba1b011c2021157ee61aa700cbecba5d35df737656 WatchSource:0}: Error finding container f03a7251428414b5ea2a6bba1b011c2021157ee61aa700cbecba5d35df737656: Status 404 returned error can't find the container with id f03a7251428414b5ea2a6bba1b011c2021157ee61aa700cbecba5d35df737656 Oct 30 21:43:01.558916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.558871 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72"] Oct 30 21:43:01.560321 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:43:01.560297 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6121b0fb_14d9_43af_a082_565deb935662.slice/crio-9a5474c8fda90f4a96932204b4e0671bb8d1511e9af66ca810404f619c34dc29 WatchSource:0}: Error finding container 9a5474c8fda90f4a96932204b4e0671bb8d1511e9af66ca810404f619c34dc29: Status 404 returned error can't find the container with id 9a5474c8fda90f4a96932204b4e0671bb8d1511e9af66ca810404f619c34dc29 Oct 30 21:43:01.646934 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.646907 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8"] Oct 30 21:43:01.647192 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:43:01.647172 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3024ff_2a7e_4083_8d14_307097d907fd.slice/crio-d0b2e0d616babab5e589ec2f4954fd59a9b535788666f1359dffcfc14a5a0b99 WatchSource:0}: Error finding container d0b2e0d616babab5e589ec2f4954fd59a9b535788666f1359dffcfc14a5a0b99: Status 404 returned error can't find the container with id d0b2e0d616babab5e589ec2f4954fd59a9b535788666f1359dffcfc14a5a0b99 Oct 30 21:43:01.649716 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.649678 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:01.649843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.649746 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:01.649843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:01.649774 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:02.220579 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220553 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220600 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220616 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" event={"ID":"cd3024ff-2a7e-4083-8d14-307097d907fd","Type":"ContainerStarted","Data":"b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8"} Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220635 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" event={"ID":"cd3024ff-2a7e-4083-8d14-307097d907fd","Type":"ContainerStarted","Data":"d0b2e0d616babab5e589ec2f4954fd59a9b535788666f1359dffcfc14a5a0b99"} Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220651 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" event={"ID":"6121b0fb-14d9-43af-a082-565deb935662","Type":"ContainerStarted","Data":"9b56d5b7bb47e4fd9fbede8b5ab4f1bfaeb1011c187244dd46c1a7b18f2b89b3"} Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220663 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" event={"ID":"6121b0fb-14d9-43af-a082-565deb935662","Type":"ContainerStarted","Data":"9a5474c8fda90f4a96932204b4e0671bb8d1511e9af66ca810404f619c34dc29"} Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220678 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" event={"ID":"0bef2b3b-edb4-4272-823f-e7c0e035f082","Type":"ContainerStarted","Data":"f425be0b4c608e4ac34d5a24744a0f469406142a4d767fd98bf599a71d64c3e3"} Oct 30 21:43:02.220994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.220696 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" event={"ID":"0bef2b3b-edb4-4272-823f-e7c0e035f082","Type":"ContainerStarted","Data":"f03a7251428414b5ea2a6bba1b011c2021157ee61aa700cbecba5d35df737656"} Oct 30 21:43:02.233034 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.233006 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:02.239377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.239339 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:02.339220 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.339166 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podStartSLOduration=2.339149258 podStartE2EDuration="2.339149258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:02.337824745 +0000 UTC m=+1534.239688812" watchObservedRunningTime="2026-10-30 21:43:02.339149258 +0000 UTC m=+1534.241013323" Oct 30 21:43:02.599985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.593182 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" podStartSLOduration=2.59316215 podStartE2EDuration="2.59316215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:02.476153441 +0000 UTC m=+1534.378017508" watchObservedRunningTime="2026-10-30 21:43:02.59316215 +0000 UTC m=+1534.495026210" Oct 30 21:43:02.604165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:02.604082 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" podStartSLOduration=3.604059026 podStartE2EDuration="3.604059026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:02.592582205 +0000 UTC m=+1534.494446315" watchObservedRunningTime="2026-10-30 21:43:02.604059026 +0000 UTC m=+1534.505923092" Oct 30 21:43:03.227496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:03.227467 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:03.228597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:03.228573 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:43:07.394110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.394025 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6"] Oct 30 21:43:07.399960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.399923 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.442380 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.442349 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2"] Oct 30 21:43:07.447138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.447113 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.500737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.500705 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6"] Oct 30 21:43:07.531385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531347 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.531589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531474 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g74\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.531589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531531 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdz5t\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.531589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531560 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.531772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531587 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531863 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.532080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.531971 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532028 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532048 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.532347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532103 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532222 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532269 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532354 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.532577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532387 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532577 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532521 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.532741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532652 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.532741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532684 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.532855 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.532711 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.558944 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.552920 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2"] Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635399 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635455 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635530 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635556 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635596 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635636 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635678 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635760 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635789 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635828 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.636304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.635941 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.637117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.636715 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.637117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.636735 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.637232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637149 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.637380 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637353 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.638619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637409 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.638619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637600 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.638619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637660 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.638619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637667 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.638619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.637909 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.638619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.638213 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.639021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.638300 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.639021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.638969 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.639142 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.639120 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g74\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.639516 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.639490 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdz5t\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.639614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.639549 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.639614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.639583 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.640805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.640779 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.642197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.642107 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.642377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.642191 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.643139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.643110 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.650193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.646769 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.695247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.695212 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.697909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.697464 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g74\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.698728 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.698696 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lbxq2\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:07.701219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.701049 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdz5t\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t\") pod \"maas-default-gateway-openshift-default-7f694df8bc-qjkt6\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.716631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.715804 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:07.762942 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:07.762906 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:08.347776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.345233 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.347776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.345337 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.347776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.345383 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.406327 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:43:08.406286 5914 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e554433_27a6_4edb_bc0e_416f50b6bb87.slice/crio-1a8a6e4e209edbed54688535cb00d023564bf0568ccfb94c62335e59271897fa\": RecentStats: unable to find data in memory cache]" Oct 30 21:43:08.480593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.480408 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6"] Oct 30 21:43:08.480851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.480833 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2"] Oct 30 21:43:08.481058 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.481033 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" event={"ID":"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9","Type":"ContainerStarted","Data":"edf8106f820bd83d0111a08706b3e68e530d746d761f15df1bb83cfe3da3baea"} Oct 30 21:43:08.502970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.502451 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.502970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.502562 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:08.502970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:08.502607 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:43:09.431023 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.430989 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" event={"ID":"3e554433-27a6-4edb-bc0e-416f50b6bb87","Type":"ContainerStarted","Data":"349b91f82e9459522ca27ebb2b44773b0797aa1ba9ae3edd0e07eb8e56e607a1"} Oct 30 21:43:09.431562 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.431298 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" event={"ID":"3e554433-27a6-4edb-bc0e-416f50b6bb87","Type":"ContainerStarted","Data":"1a8a6e4e209edbed54688535cb00d023564bf0568ccfb94c62335e59271897fa"} Oct 30 21:43:09.434125 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.434097 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" event={"ID":"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9","Type":"ContainerStarted","Data":"ca769f274d18fec1e8bfbeaeb1cbe0dde89fb18c5ed6f7eaf0c2e909276ea8c1"} Oct 30 21:43:09.722173 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.721081 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:09.724501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.724265 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:09.724501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.724328 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:09.764014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.763982 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:09.765084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.765059 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:09.765182 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.765127 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:09.830795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:09.830666 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podStartSLOduration=2.830645237 podStartE2EDuration="2.830645237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.6681572 +0000 UTC m=+1541.570021264" watchObservedRunningTime="2026-10-30 21:43:09.830645237 +0000 UTC m=+1541.732509302" Oct 30 21:43:10.716909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:10.716573 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:10.716909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:10.716648 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:10.763762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:10.763730 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:10.763959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:10.763793 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:11.723907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:11.723374 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:11.723907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:11.723437 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:11.764386 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:11.764348 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:11.764555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:11.764412 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:12.351964 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:43:12.351923 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:43:12.725485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:12.725022 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:12.725485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:12.725094 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:12.763843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:12.763807 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:12.764023 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:12.763859 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:13.720435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:13.720394 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:13.720739 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:13.720705 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:13.763539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:13.763504 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:13.763978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:13.763557 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:14.236725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.236699 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:43:14.236929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.236780 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:43:14.372708 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.372659 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podStartSLOduration=7.37264543 podStartE2EDuration="7.37264543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:09.807379211 +0000 UTC m=+1541.709243277" watchObservedRunningTime="2026-10-30 21:43:14.37264543 +0000 UTC m=+1546.274509496" Oct 30 21:43:14.722909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.719348 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:14.722909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.719431 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:14.763802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.763764 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:14.764205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:14.763815 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:15.716561 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:15.716517 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:43:15.716722 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:15.716591 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 21:43:15.763524 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:15.763488 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 21:43:15.763680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:15.763571 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 21:43:17.370685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.370576 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:17.372239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.371834 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:17.373568 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.373540 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:43:17.463061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.463010 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:17.464004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.463982 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:17.466513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.466489 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:43:17.834428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.834391 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x"] Oct 30 21:43:17.840513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.840486 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:17.896752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.896719 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x"] Oct 30 21:43:17.898453 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.898430 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3b21b4-serving-cert\"" Oct 30 21:43:17.932534 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.932464 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:17.932967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.932928 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:17.933212 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.933175 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:17.934269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:17.934240 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gppj7\" (UniqueName: \"kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.036179 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.036146 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gppj7\" (UniqueName: \"kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.036382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.036256 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.036382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.036304 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.036382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.036333 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.036552 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:43:18.036440 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-serving-cert: secret "maas-api-e2e-shared-a-w0-3b21b4-serving-cert" not found Oct 30 21:43:18.036552 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:43:18.036513 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls podName:6df0ea15-7904-44df-9d92-2751c2f9edd4 nodeName:}" failed. No retries permitted until 2026-10-30 21:43:18.536493529 +0000 UTC m=+1550.438357575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls") pod "maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" (UID: "6df0ea15-7904-44df-9d92-2751c2f9edd4") : secret "maas-api-e2e-shared-a-w0-3b21b4-serving-cert" not found Oct 30 21:43:18.037050 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.037024 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.039061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.039040 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.122046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.116996 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gppj7\" (UniqueName: \"kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.576084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.576046 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.578724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.578702 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:18.767775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:18.767738 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:19.152976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:19.152844 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x"] Oct 30 21:43:19.158434 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:43:19.158400 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df0ea15_7904_44df_9d92_2751c2f9edd4.slice/crio-e48310eee33bc631e6355cd05593720bec5cd0444339582220242d342543b9e7 WatchSource:0}: Error finding container e48310eee33bc631e6355cd05593720bec5cd0444339582220242d342543b9e7: Status 404 returned error can't find the container with id e48310eee33bc631e6355cd05593720bec5cd0444339582220242d342543b9e7 Oct 30 21:43:19.600176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:19.600091 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" event={"ID":"6df0ea15-7904-44df-9d92-2751c2f9edd4","Type":"ContainerStarted","Data":"e48310eee33bc631e6355cd05593720bec5cd0444339582220242d342543b9e7"} Oct 30 21:43:20.609360 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:20.609311 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" event={"ID":"6df0ea15-7904-44df-9d92-2751c2f9edd4","Type":"ContainerStarted","Data":"c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f"} Oct 30 21:43:20.609812 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:20.609384 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:20.740069 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:20.740022 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" podStartSLOduration=2.950124098 podStartE2EDuration="3.740007093s" totalImagesPullingTime="789.882995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:20.738929429 +0000 UTC m=+1552.640793495" watchObservedRunningTime="2026-10-30 21:43:20.740007093 +0000 UTC m=+1552.641871157" Oct 30 21:43:26.618346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:26.618301 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.94:8443/health\": dial tcp 10.134.0.94:8443: connect: connection refused" Oct 30 21:43:30.688137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:30.688100 5914 generic.go:356] "Generic (PLEG): container finished" podID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerID="c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f" exitCode=1 Oct 30 21:43:30.688622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:30.688176 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" event={"ID":"6df0ea15-7904-44df-9d92-2751c2f9edd4","Type":"ContainerDied","Data":"c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f"} Oct 30 21:43:30.688683 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:30.688643 5914 scope.go:122] "RemoveContainer" containerID="c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f" Oct 30 21:43:32.749962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:32.749920 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" event={"ID":"6df0ea15-7904-44df-9d92-2751c2f9edd4","Type":"ContainerStarted","Data":"cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283"} Oct 30 21:43:32.752197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:32.752076 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:38.768673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:38.768644 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:43:42.346298 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:43:42.346252 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:43:51.815252 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:51.815216 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:43:58.543919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.533610 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7"] Oct 30 21:43:58.639128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.639092 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7"] Oct 30 21:43:58.639449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.639431 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.672875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.672845 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.672875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.672917 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.673155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.672983 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zss9q\" (UniqueName: \"kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.673155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.673051 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.698362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.698330 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1915737c-serving-cert\"" Oct 30 21:43:58.800647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.800538 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.800647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.800591 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.800939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.800664 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zss9q\" (UniqueName: \"kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.800939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.800740 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.802515 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.802481 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.804382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.804357 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.804582 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.804559 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.860858 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.860825 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zss9q\" (UniqueName: \"kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q\") pod \"maas-api-e2e-ait-1915737c-757ff464cc-887p7\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:58.966346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:58.966302 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:43:59.487661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:43:59.487632 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7"] Oct 30 21:43:59.488229 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:43:59.488203 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda59ea7f_dd01_46d5_8bd9_f4f4a259a2c6.slice/crio-f7282428a156e39faf17879f30f0103e852c93252a707e5deed4293e2ff010e4 WatchSource:0}: Error finding container f7282428a156e39faf17879f30f0103e852c93252a707e5deed4293e2ff010e4: Status 404 returned error can't find the container with id f7282428a156e39faf17879f30f0103e852c93252a707e5deed4293e2ff010e4 Oct 30 21:44:00.249746 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:00.249715 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" event={"ID":"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6","Type":"ContainerStarted","Data":"f7282428a156e39faf17879f30f0103e852c93252a707e5deed4293e2ff010e4"} Oct 30 21:44:01.259580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:01.259537 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" event={"ID":"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6","Type":"ContainerStarted","Data":"78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc"} Oct 30 21:44:01.260033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:01.259614 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:44:01.419678 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:01.419618 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" podStartSLOduration=2.625544234 podStartE2EDuration="3.419605228s" totalImagesPullingTime="794.060994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:01.374971243 +0000 UTC m=+1593.276835309" watchObservedRunningTime="2026-10-30 21:44:01.419605228 +0000 UTC m=+1593.321469292" Oct 30 21:44:02.393035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:02.392998 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:44:02.393529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:02.393061 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:04.285604 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:04.285564 5914 generic.go:356] "Generic (PLEG): container finished" podID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerID="04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b" exitCode=0 Oct 30 21:44:04.286089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:04.285643 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" event={"ID":"81bbf7f4-758d-497d-9bfe-a4273c871f63","Type":"ContainerDied","Data":"04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b"} Oct 30 21:44:04.286140 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:04.286107 5914 scope.go:122] "RemoveContainer" containerID="04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b" Oct 30 21:44:04.287689 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:04.287659 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:04.287805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:04.287739 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:04.287805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:04.287791 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:05.306848 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:05.306807 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" event={"ID":"81bbf7f4-758d-497d-9bfe-a4273c871f63","Type":"ContainerStarted","Data":"11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818"} Oct 30 21:44:05.351714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:05.351676 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:44:05.351996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:05.351971 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:44:05.352062 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:05.352022 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 21:44:06.352409 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:06.352368 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:44:06.352947 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:06.352455 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 21:44:07.367287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:07.366748 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:44:07.367287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:07.366844 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 21:44:07.385127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:07.385091 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:44:08.154909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.136265 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh"] Oct 30 21:44:08.197957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.179226 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.220141 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.220102 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh"] Oct 30 21:44:08.303374 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.303322 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305095 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305187 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305281 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305472 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305521 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305578 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.305705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305696 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpdx\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.306199 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.305757 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.353005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.352967 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:44:08.353203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.353024 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 21:44:08.407931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.407799 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.407931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.407865 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.407931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.407927 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.407965 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408052 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpdx\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408103 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408197 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408307 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408346 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408383 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.408546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.408401 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.411961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.411917 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.412486 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.412457 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.414517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.414496 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.414693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.414675 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.415263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.415240 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.468953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.468913 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.471209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.471184 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpdx\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-fzjdh\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:08.598276 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:08.598236 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:09.039025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.038997 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh"] Oct 30 21:44:09.039298 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:44:09.039276 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b30a216_a370_463e_b9c9_125427eec2fb.slice/crio-4ce0d8589ec3dea161894111e342695614744602e2a1dccc5a34eefc59017172 WatchSource:0}: Error finding container 4ce0d8589ec3dea161894111e342695614744602e2a1dccc5a34eefc59017172: Status 404 returned error can't find the container with id 4ce0d8589ec3dea161894111e342695614744602e2a1dccc5a34eefc59017172 Oct 30 21:44:09.042226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.042178 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:09.042304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.042270 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:09.042350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.042301 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:09.351861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.351769 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:44:09.351861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.351840 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 21:44:09.407787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.406670 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" event={"ID":"5b30a216-a370-463e-b9c9-125427eec2fb","Type":"ContainerStarted","Data":"0d730ef5c4c889cdd5ef8e3eaec5d035bab931772717db13f39f0f001da32aa3"} Oct 30 21:44:09.407787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.406734 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" event={"ID":"5b30a216-a370-463e-b9c9-125427eec2fb","Type":"ContainerStarted","Data":"4ce0d8589ec3dea161894111e342695614744602e2a1dccc5a34eefc59017172"} Oct 30 21:44:09.593647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.593571 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podStartSLOduration=2.593545522 podStartE2EDuration="2.593545522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:09.58530578 +0000 UTC m=+1601.487169846" watchObservedRunningTime="2026-10-30 21:44:09.593545522 +0000 UTC m=+1601.495409587" Oct 30 21:44:09.599522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:09.599486 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:10.352488 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:10.352451 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:44:10.352696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:10.352513 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 21:44:10.601098 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:10.601054 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:10.601520 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:10.601134 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:11.357132 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:11.357098 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:44:11.357414 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:11.357393 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:44:11.358378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:11.358357 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:44:11.599368 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:11.599323 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:11.599558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:11.599400 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:12.347253 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:44:12.347208 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:44:12.599460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:12.599353 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:12.599460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:12.599415 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:13.599390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:13.599348 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:13.599851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:13.599406 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:14.612916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:14.611858 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:14.612916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:14.611953 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:15.602256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:15.602221 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:15.602578 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:15.602533 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:16.613045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:16.613008 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:16.615130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:16.615092 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:17.603038 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:17.603000 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:17.603282 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:17.603250 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:18.601479 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:18.601427 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-fzjdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:44:18.602096 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:18.601509 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 21:44:20.302413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:20.302377 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:20.303804 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:20.303776 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:20.304831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:20.304810 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:44:26.007093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.007050 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:44:26.007569 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.007137 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:26.684992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.684960 5914 generic.go:356] "Generic (PLEG): container finished" podID="1e095191-51ca-4537-8220-6bc220e6abce" containerID="4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993" exitCode=0 Oct 30 21:44:26.685226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.685036 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" event={"ID":"1e095191-51ca-4537-8220-6bc220e6abce","Type":"ContainerDied","Data":"4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993"} Oct 30 21:44:26.685608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.685588 5914 scope.go:122] "RemoveContainer" containerID="4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993" Oct 30 21:44:26.687322 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.687293 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:26.687430 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.687361 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:26.687430 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:26.687395 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:44:27.707446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:27.707411 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" event={"ID":"1e095191-51ca-4537-8220-6bc220e6abce","Type":"ContainerStarted","Data":"a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f"} Oct 30 21:44:28.569573 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:28.569538 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:44:28.569991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:28.569963 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:28.570127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:28.570034 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:29.569766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:29.569729 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:29.570276 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:29.569800 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:30.569613 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:30.569575 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:30.569798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:30.569638 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:31.569801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:31.569765 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:31.570235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:31.569827 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:32.570004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:32.569952 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 21:44:32.570507 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:32.570028 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 21:44:33.574382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:33.574348 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:44:33.574793 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:33.574776 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:44:33.575574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:33.575553 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:44:42.347649 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:44:42.347565 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:44:52.913358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:44:52.913328 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:45:00.344533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.344490 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x"] Oct 30 21:45:00.351374 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.351345 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.405963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.405934 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x"] Oct 30 21:45:00.496992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.496949 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 21:45:00.524578 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.524532 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.524759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.524736 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79d6l\" (UniqueName: \"kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.625744 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.625715 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79d6l\" (UniqueName: \"kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.625948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.625917 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.626538 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.626510 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.685235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.685195 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79d6l\" (UniqueName: \"kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l\") pod \"maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:00.927745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.927662 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72"] Oct 30 21:45:00.967544 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:00.967515 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:01.048966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:01.048375 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs"] Oct 30 21:45:01.396668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:01.389146 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x"] Oct 30 21:45:02.326809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:02.326771 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerStarted","Data":"5550f89a5fc9b54529d4c45b873d25cc0843154087d499226b9bab6290aa74b8"} Oct 30 21:45:05.703812 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.703770 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g"] Oct 30 21:45:05.709795 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.709767 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:05.761893 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.761853 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-29b7ead1-serving-cert\"" Oct 30 21:45:05.804749 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.804714 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g"] Oct 30 21:45:05.883906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.873530 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:05.883906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.873687 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29x8n\" (UniqueName: \"kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:05.883906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.873743 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:05.906321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.906277 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:05.927976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:05.927931 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" podUID="6121b0fb-14d9-43af-a082-565deb935662" containerName="payload-processing" containerID="cri-o://9b56d5b7bb47e4fd9fbede8b5ab4f1bfaeb1011c187244dd46c1a7b18f2b89b3" gracePeriod=25 Oct 30 21:45:06.014624 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.014577 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29x8n\" (UniqueName: \"kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.014794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.014669 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.014794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.014784 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.015066 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.015044 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.018707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.018679 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.019416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.019393 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.019694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.019663 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.048713 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.048647 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" podUID="0bef2b3b-edb4-4272-823f-e7c0e035f082" containerName="payload-pre-processing" containerID="cri-o://f425be0b4c608e4ac34d5a24744a0f469406142a4d767fd98bf599a71d64c3e3" gracePeriod=25 Oct 30 21:45:06.129931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.126613 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29x8n\" (UniqueName: \"kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n\") pod \"maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.425255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.425159 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:06.437319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.437088 5914 generic.go:356] "Generic (PLEG): container finished" podID="6121b0fb-14d9-43af-a082-565deb935662" containerID="9b56d5b7bb47e4fd9fbede8b5ab4f1bfaeb1011c187244dd46c1a7b18f2b89b3" exitCode=0 Oct 30 21:45:06.437319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.437241 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" event={"ID":"6121b0fb-14d9-43af-a082-565deb935662","Type":"ContainerDied","Data":"9b56d5b7bb47e4fd9fbede8b5ab4f1bfaeb1011c187244dd46c1a7b18f2b89b3"} Oct 30 21:45:06.532479 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.532443 5914 generic.go:356] "Generic (PLEG): container finished" podID="0bef2b3b-edb4-4272-823f-e7c0e035f082" containerID="f425be0b4c608e4ac34d5a24744a0f469406142a4d767fd98bf599a71d64c3e3" exitCode=0 Oct 30 21:45:06.532668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.532560 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" event={"ID":"0bef2b3b-edb4-4272-823f-e7c0e035f082","Type":"ContainerDied","Data":"f425be0b4c608e4ac34d5a24744a0f469406142a4d767fd98bf599a71d64c3e3"} Oct 30 21:45:06.536977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.535303 5914 generic.go:356] "Generic (PLEG): container finished" podID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerID="a138ea3d403cf5b43b46b2fefd45b3d3fea82629028c5ff719c2c615cfc6b6dd" exitCode=60 Oct 30 21:45:06.536977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.535508 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerDied","Data":"a138ea3d403cf5b43b46b2fefd45b3d3fea82629028c5ff719c2c615cfc6b6dd"} Oct 30 21:45:06.536977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.536038 5914 scope.go:122] "RemoveContainer" containerID="a138ea3d403cf5b43b46b2fefd45b3d3fea82629028c5ff719c2c615cfc6b6dd" Oct 30 21:45:06.643162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.643139 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:45:06.714394 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.714360 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p\" (UniqueName: \"kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p\") pod \"6121b0fb-14d9-43af-a082-565deb935662\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " Oct 30 21:45:06.714858 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.714720 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume\") pod \"6121b0fb-14d9-43af-a082-565deb935662\" (UID: \"6121b0fb-14d9-43af-a082-565deb935662\") " Oct 30 21:45:06.715777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.715747 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume" pod "6121b0fb-14d9-43af-a082-565deb935662" (UID: "6121b0fb-14d9-43af-a082-565deb935662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:06.726919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.722588 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p" pod "6121b0fb-14d9-43af-a082-565deb935662" (UID: "6121b0fb-14d9-43af-a082-565deb935662"). InnerVolumeSpecName "kube-api-access-wcd7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:06.766236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.766200 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:45:06.816258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.816211 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm\" (UniqueName: \"kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm\") pod \"0bef2b3b-edb4-4272-823f-e7c0e035f082\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " Oct 30 21:45:06.816472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.816453 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume\") pod \"0bef2b3b-edb4-4272-823f-e7c0e035f082\" (UID: \"0bef2b3b-edb4-4272-823f-e7c0e035f082\") " Oct 30 21:45:06.817352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.817321 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6121b0fb-14d9-43af-a082-565deb935662-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:06.817453 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.817356 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcd7p\" (UniqueName: \"kubernetes.io/projected/6121b0fb-14d9-43af-a082-565deb935662-kube-api-access-wcd7p\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:06.817747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.817721 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume" pod "0bef2b3b-edb4-4272-823f-e7c0e035f082" (UID: "0bef2b3b-edb4-4272-823f-e7c0e035f082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:06.818960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.818940 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm" pod "0bef2b3b-edb4-4272-823f-e7c0e035f082" (UID: "0bef2b3b-edb4-4272-823f-e7c0e035f082"). InnerVolumeSpecName "kube-api-access-tqgnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:06.918925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.918870 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bef2b3b-edb4-4272-823f-e7c0e035f082-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:06.918925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:06.918927 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqgnm\" (UniqueName: \"kubernetes.io/projected/0bef2b3b-edb4-4272-823f-e7c0e035f082-kube-api-access-tqgnm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:07.012104 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:45:07.012065 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc1d63f_f5db_4727_bee9_5bee78d38e8a.slice/crio-ba7cbd3bc9514273c6983f324991d720339d82ac247875db9fd52446aca67f4d WatchSource:0}: Error finding container ba7cbd3bc9514273c6983f324991d720339d82ac247875db9fd52446aca67f4d: Status 404 returned error can't find the container with id ba7cbd3bc9514273c6983f324991d720339d82ac247875db9fd52446aca67f4d Oct 30 21:45:07.015651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.015627 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g"] Oct 30 21:45:07.547148 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.547113 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" event={"ID":"6cc1d63f-f5db-4727-bee9-5bee78d38e8a","Type":"ContainerStarted","Data":"ba7cbd3bc9514273c6983f324991d720339d82ac247875db9fd52446aca67f4d"} Oct 30 21:45:07.548693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.548669 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" Oct 30 21:45:07.548852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.548697 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72" event={"ID":"6121b0fb-14d9-43af-a082-565deb935662","Type":"ContainerDied","Data":"9a5474c8fda90f4a96932204b4e0671bb8d1511e9af66ca810404f619c34dc29"} Oct 30 21:45:07.548852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.548810 5914 scope.go:122] "RemoveContainer" containerID="9b56d5b7bb47e4fd9fbede8b5ab4f1bfaeb1011c187244dd46c1a7b18f2b89b3" Oct 30 21:45:07.550166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.550143 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" event={"ID":"0bef2b3b-edb4-4272-823f-e7c0e035f082","Type":"ContainerDied","Data":"f03a7251428414b5ea2a6bba1b011c2021157ee61aa700cbecba5d35df737656"} Oct 30 21:45:07.550270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.550208 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs" Oct 30 21:45:07.553407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.553383 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerStarted","Data":"e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8"} Oct 30 21:45:07.560638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.560614 5914 scope.go:122] "RemoveContainer" containerID="f425be0b4c608e4ac34d5a24744a0f469406142a4d767fd98bf599a71d64c3e3" Oct 30 21:45:07.717859 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.717823 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72"] Oct 30 21:45:07.721536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.721512 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9173600a-6bbdfc7b89-jkf72"] Oct 30 21:45:07.835191 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:07.835145 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" podStartSLOduration=4.343119937 podStartE2EDuration="7.83513122s" totalImagesPullingTime="3.492011283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:07.83491125 +0000 UTC m=+1659.736775315" watchObservedRunningTime="2026-10-30 21:45:07.83513122 +0000 UTC m=+1659.736995284" Oct 30 21:45:08.010700 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.003204 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs"] Oct 30 21:45:08.031057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.030998 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9173600a-5d8c7564d7-j7lzs"] Oct 30 21:45:08.218862 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.218827 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bef2b3b-edb4-4272-823f-e7c0e035f082" path="/var/lib/kubelet/pods/0bef2b3b-edb4-4272-823f-e7c0e035f082/volumes" Oct 30 21:45:08.219326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.219309 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6121b0fb-14d9-43af-a082-565deb935662" path="/var/lib/kubelet/pods/6121b0fb-14d9-43af-a082-565deb935662/volumes" Oct 30 21:45:08.564684 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.564651 5914 generic.go:356] "Generic (PLEG): container finished" podID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerID="e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8" exitCode=60 Oct 30 21:45:08.564838 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.564727 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerDied","Data":"e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8"} Oct 30 21:45:08.564838 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.564788 5914 scope.go:122] "RemoveContainer" containerID="a138ea3d403cf5b43b46b2fefd45b3d3fea82629028c5ff719c2c615cfc6b6dd" Oct 30 21:45:08.565176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.565159 5914 scope.go:122] "RemoveContainer" containerID="e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8" Oct 30 21:45:08.565608 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:08.565584 5914 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x_odh-ai-gateway-infra(ee5f0ea6-b5da-4349-b24f-f137c82de829)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" Oct 30 21:45:08.566739 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.566715 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" event={"ID":"6cc1d63f-f5db-4727-bee9-5bee78d38e8a","Type":"ContainerStarted","Data":"e6ae3607f1781f614b68984c30f25cb9ebb22cf99c28f780eab1cf3d4de32323"} Oct 30 21:45:08.566862 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.566843 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:08.809856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:08.809799 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" podStartSLOduration=2.853745682 podStartE2EDuration="3.80978384s" totalImagesPullingTime="956.038158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:08.806397038 +0000 UTC m=+1660.708261097" watchObservedRunningTime="2026-10-30 21:45:08.80978384 +0000 UTC m=+1660.711647902" Oct 30 21:45:09.602528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:09.602462 5914 scope.go:122] "RemoveContainer" containerID="e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8" Oct 30 21:45:09.602815 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:09.602786 5914 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x_odh-ai-gateway-infra(ee5f0ea6-b5da-4349-b24f-f137c82de829)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" Oct 30 21:45:12.346398 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:12.346350 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:45:14.610472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:14.610436 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:45:15.649684 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:15.649650 5914 generic.go:356] "Generic (PLEG): container finished" podID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerID="b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8" exitCode=0 Oct 30 21:45:15.650133 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:15.649730 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" event={"ID":"cd3024ff-2a7e-4083-8d14-307097d907fd","Type":"ContainerDied","Data":"b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8"} Oct 30 21:45:15.650275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:15.650260 5914 scope.go:122] "RemoveContainer" containerID="b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8" Oct 30 21:45:15.652070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:15.652040 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:15.652134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:15.652111 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:15.652170 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:15.652149 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:45:16.658775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:16.658739 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" event={"ID":"cd3024ff-2a7e-4083-8d14-307097d907fd","Type":"ContainerStarted","Data":"0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8"} Oct 30 21:45:17.234082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:17.234049 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:45:17.235253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:17.235228 5914 patch_prober.go:29] interesting pod/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:45:17.235350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:17.235276 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:45:18.227989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:18.227949 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:45:18.233393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:18.233358 5914 patch_prober.go:29] interesting pod/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:45:18.233527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:18.233411 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:45:19.233416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:19.233382 5914 patch_prober.go:29] interesting pod/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:45:19.233930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:19.233440 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:45:20.233408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:20.233275 5914 patch_prober.go:29] interesting pod/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:45:20.233627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:20.233440 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:45:21.234357 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:21.234325 5914 patch_prober.go:29] interesting pod/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:45:21.234748 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:21.234380 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:45:22.210117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:22.210080 5914 scope.go:122] "RemoveContainer" containerID="e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8" Oct 30 21:45:22.233258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:22.233226 5914 patch_prober.go:29] interesting pod/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:45:22.233418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:22.233286 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 21:45:22.893722 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:22.893683 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerStarted","Data":"6894101b2abe06364ec42d389a38492e352002267d486244c77ae1acbdc53ab9"} Oct 30 21:45:23.238192 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:23.238114 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:45:23.239345 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:23.239321 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:45:23.902419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:23.902383 5914 generic.go:356] "Generic (PLEG): container finished" podID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerID="6894101b2abe06364ec42d389a38492e352002267d486244c77ae1acbdc53ab9" exitCode=60 Oct 30 21:45:23.903036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:23.902459 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerDied","Data":"6894101b2abe06364ec42d389a38492e352002267d486244c77ae1acbdc53ab9"} Oct 30 21:45:23.903036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:23.902526 5914 scope.go:122] "RemoveContainer" containerID="e069c8cd36752c3c400f34faa6e3898ecc59eaeb1c3143e5ca1dde84886f14b8" Oct 30 21:45:23.903036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:23.903009 5914 scope.go:122] "RemoveContainer" containerID="6894101b2abe06364ec42d389a38492e352002267d486244c77ae1acbdc53ab9" Oct 30 21:45:23.903459 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:23.903432 5914 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x_odh-ai-gateway-infra(ee5f0ea6-b5da-4349-b24f-f137c82de829)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" Oct 30 21:45:24.021831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:24.021800 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x"] Oct 30 21:45:25.100798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.100767 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:25.162692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.162606 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca\") pod \"ee5f0ea6-b5da-4349-b24f-f137c82de829\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " Oct 30 21:45:25.162692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.162656 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l\" (UniqueName: \"kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l\") pod \"ee5f0ea6-b5da-4349-b24f-f137c82de829\" (UID: \"ee5f0ea6-b5da-4349-b24f-f137c82de829\") " Oct 30 21:45:25.163114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.163093 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca" pod "ee5f0ea6-b5da-4349-b24f-f137c82de829" (UID: "ee5f0ea6-b5da-4349-b24f-f137c82de829"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:25.163231 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.163219 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f0ea6-b5da-4349-b24f-f137c82de829-maas-api-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.165005 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.164988 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l" pod "ee5f0ea6-b5da-4349-b24f-f137c82de829" (UID: "ee5f0ea6-b5da-4349-b24f-f137c82de829"). InnerVolumeSpecName "kube-api-access-79d6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:25.264774 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.264742 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79d6l\" (UniqueName: \"kubernetes.io/projected/ee5f0ea6-b5da-4349-b24f-f137c82de829-kube-api-access-79d6l\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:45:25.921056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.921021 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" event={"ID":"ee5f0ea6-b5da-4349-b24f-f137c82de829","Type":"ContainerDied","Data":"5550f89a5fc9b54529d4c45b873d25cc0843154087d499226b9bab6290aa74b8"} Oct 30 21:45:25.921056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.921038 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x" Oct 30 21:45:25.921283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:25.921085 5914 scope.go:122] "RemoveContainer" containerID="6894101b2abe06364ec42d389a38492e352002267d486244c77ae1acbdc53ab9" Oct 30 21:45:26.085704 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:26.085666 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x"] Oct 30 21:45:26.135619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:26.135570 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-07fa8e-29889945-cjv6x"] Oct 30 21:45:26.218906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:26.218813 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" path="/var/lib/kubelet/pods/ee5f0ea6-b5da-4349-b24f-f137c82de829/volumes" Oct 30 21:45:42.347661 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:42.347622 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:45:44.419782 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.419738 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t"] Oct 30 21:45:44.420519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420500 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.420519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420518 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420537 5914 cpu_manager.go:414] "containerMap: removing container" podUID="0bef2b3b-edb4-4272-823f-e7c0e035f082" containerName="payload-pre-processing" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420542 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bef2b3b-edb4-4272-823f-e7c0e035f082" containerName="payload-pre-processing" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420551 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420555 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420574 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420579 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420592 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6121b0fb-14d9-43af-a082-565deb935662" containerName="payload-processing" Oct 30 21:45:44.420668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420596 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6121b0fb-14d9-43af-a082-565deb935662" containerName="payload-processing" Oct 30 21:45:44.421136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420694 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.421136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420704 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bef2b3b-edb4-4272-823f-e7c0e035f082" containerName="payload-pre-processing" Oct 30 21:45:44.421136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420711 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:45:44.421136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.420720 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6121b0fb-14d9-43af-a082-565deb935662" containerName="payload-processing" Oct 30 21:45:44.425806 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.425789 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.437389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.437357 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fsh\" (UniqueName: \"kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.437601 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.437579 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.437723 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.437707 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.437819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.437802 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.478186 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.478155 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-d6a241-serving-cert\"" Oct 30 21:45:44.532722 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.532689 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t"] Oct 30 21:45:44.540670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.540576 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.540952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.540701 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.540952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.540849 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.541093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.540977 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fsh\" (UniqueName: \"kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.541404 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:44.541382 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-serving-cert: secret "maas-api-e2e-shared-b-w0-d6a241-serving-cert" not found Oct 30 21:45:44.541531 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:45:44.541453 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls podName:a1de05e3-8a4c-4900-a123-0f83a2fd06aa nodeName:}" failed. No retries permitted until 2026-10-30 21:45:45.0414327 +0000 UTC m=+1696.943296745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls") pod "maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" (UID: "a1de05e3-8a4c-4900-a123-0f83a2fd06aa") : secret "maas-api-e2e-shared-b-w0-d6a241-serving-cert" not found Oct 30 21:45:44.542278 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.542258 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.545733 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.545713 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:44.599555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:44.599519 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fsh\" (UniqueName: \"kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:45.084483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:45.084450 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:45.087186 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:45.087167 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:45.341833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:45.341736 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:45.726175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:45.726142 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t"] Oct 30 21:45:45.726594 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:45:45.726407 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1de05e3_8a4c_4900_a123_0f83a2fd06aa.slice/crio-62d417a50cb5ad93e624d7fd2ea8496d08c555ff7443766afe867b5d1dfc481d WatchSource:0}: Error finding container 62d417a50cb5ad93e624d7fd2ea8496d08c555ff7443766afe867b5d1dfc481d: Status 404 returned error can't find the container with id 62d417a50cb5ad93e624d7fd2ea8496d08c555ff7443766afe867b5d1dfc481d Oct 30 21:45:46.089511 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:46.089417 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" event={"ID":"a1de05e3-8a4c-4900-a123-0f83a2fd06aa","Type":"ContainerStarted","Data":"62d417a50cb5ad93e624d7fd2ea8496d08c555ff7443766afe867b5d1dfc481d"} Oct 30 21:45:47.099081 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:47.099042 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" event={"ID":"a1de05e3-8a4c-4900-a123-0f83a2fd06aa","Type":"ContainerStarted","Data":"bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be"} Oct 30 21:45:47.099513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:47.099223 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:47.224261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:47.224208 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" podStartSLOduration=2.348350862 podStartE2EDuration="3.224193518s" totalImagesPullingTime="875.842656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:47.219998737 +0000 UTC m=+1699.121862823" watchObservedRunningTime="2026-10-30 21:45:47.224193518 +0000 UTC m=+1699.126057608" Oct 30 21:45:47.713676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:47.713645 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:45:53.112521 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:53.112488 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:45:57.133726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.133685 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc"] Oct 30 21:45:57.134302 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.133979 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" containerID="cri-o://01f7786027aec1d1333f97ea40671b4d741c808ba713583937a2465675510c60" gracePeriod=30 Oct 30 21:45:57.183416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.183382 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7"] Oct 30 21:45:57.183663 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.183624 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" containerID="cri-o://e83b3d0ea486304f6787b4dbafec18c16b48c7bba7331497992db48bfe6363a3" gracePeriod=30 Oct 30 21:45:57.187312 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.187282 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x"] Oct 30 21:45:57.187493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.187472 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" containerID="cri-o://e1084a421bde95c150fb3e2b2b22a87cbf677144b712b2f0f445bf6f9de536ca" gracePeriod=30 Oct 30 21:45:57.233112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.233079 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb"] Oct 30 21:45:57.233341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:45:57.233317 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" containerID="cri-o://429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8" gracePeriod=30 Oct 30 21:46:01.475527 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:01.475487 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:01.475956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:01.475545 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:01.968972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:01.968930 5914 patch_prober.go:29] interesting pod/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:01.969179 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:01.968997 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:02.319815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.319779 5914 generic.go:356] "Generic (PLEG): container finished" podID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerID="e83b3d0ea486304f6787b4dbafec18c16b48c7bba7331497992db48bfe6363a3" exitCode=0 Oct 30 21:46:02.320073 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.319854 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" event={"ID":"0900c473-515d-44c1-8ce6-06d8e9f4d350","Type":"ContainerDied","Data":"e83b3d0ea486304f6787b4dbafec18c16b48c7bba7331497992db48bfe6363a3"} Oct 30 21:46:02.321847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.321815 5914 generic.go:356] "Generic (PLEG): container finished" podID="8681933e-71c3-44b2-ba83-71d851812d50" containerID="e1084a421bde95c150fb3e2b2b22a87cbf677144b712b2f0f445bf6f9de536ca" exitCode=0 Oct 30 21:46:02.321971 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.321925 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" event={"ID":"8681933e-71c3-44b2-ba83-71d851812d50","Type":"ContainerDied","Data":"e1084a421bde95c150fb3e2b2b22a87cbf677144b712b2f0f445bf6f9de536ca"} Oct 30 21:46:02.323837 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.323815 5914 generic.go:356] "Generic (PLEG): container finished" podID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerID="01f7786027aec1d1333f97ea40671b4d741c808ba713583937a2465675510c60" exitCode=0 Oct 30 21:46:02.323942 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.323918 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" event={"ID":"502d78a3-3a23-4a04-87f0-89aa189dd010","Type":"ContainerDied","Data":"01f7786027aec1d1333f97ea40671b4d741c808ba713583937a2465675510c60"} Oct 30 21:46:02.519443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.519349 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95"] Oct 30 21:46:02.520356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.520317 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" containerID="cri-o://a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f" gracePeriod=30 Oct 30 21:46:02.585270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.584607 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:46:02.658024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.657751 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.657797 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.657847 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.658011 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.658097 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.658136 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.658204 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.658359 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.658478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.658423 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs\") pod \"502d78a3-3a23-4a04-87f0-89aa189dd010\" (UID: \"502d78a3-3a23-4a04-87f0-89aa189dd010\") " Oct 30 21:46:02.659661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.659633 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.661421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.661399 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:02.662531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.662200 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.662531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.662367 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.662801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.662772 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.663266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.663242 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.669653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.667019 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.669653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.667169 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "kube-api-access-84x4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.669653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.669244 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo" pod "502d78a3-3a23-4a04-87f0-89aa189dd010" (UID: "502d78a3-3a23-4a04-87f0-89aa189dd010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:02.674009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.673734 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:46:02.679820 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.679793 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:46:02.726687 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.726659 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:46:02.759970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.759932 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.759999 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760036 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760083 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760113 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760183 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760262 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760298 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760292 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.760581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760391 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760438 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760509 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.760581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760514 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760579 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760686 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760778 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760768 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760810 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.760958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760850 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.760958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760927 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.760958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760944 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket\") pod \"0900c473-515d-44c1-8ce6-06d8e9f4d350\" (UID: \"0900c473-515d-44c1-8ce6-06d8e9f4d350\") " Oct 30 21:46:02.761151 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.760984 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs\") pod \"8681933e-71c3-44b2-ba83-71d851812d50\" (UID: \"8681933e-71c3-44b2-ba83-71d851812d50\") " Oct 30 21:46:02.761203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.761181 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.761726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.761467 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:02.762255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762227 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/502d78a3-3a23-4a04-87f0-89aa189dd010-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762262 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762278 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762294 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762309 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762323 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762336 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762350 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.762370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762365 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762379 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762395 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0900c473-515d-44c1-8ce6-06d8e9f4d350-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762409 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/502d78a3-3a23-4a04-87f0-89aa189dd010-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762423 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84x4d\" (UniqueName: \"kubernetes.io/projected/502d78a3-3a23-4a04-87f0-89aa189dd010-kube-api-access-84x4d\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762438 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762684 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.763024 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.762949 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.763399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.763146 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.763399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.763388 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.763734 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.763705 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:02.764857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.764836 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.774752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.765693 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "kube-api-access-ptq9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.774752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.768630 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:02.774752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.769505 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.774752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.769598 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:02.774752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.769695 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.774752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.769790 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg" pod "8681933e-71c3-44b2-ba83-71d851812d50" (UID: "8681933e-71c3-44b2-ba83-71d851812d50"). InnerVolumeSpecName "kube-api-access-ggzlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.786430 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.786173 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy" pod "0900c473-515d-44c1-8ce6-06d8e9f4d350" (UID: "0900c473-515d-44c1-8ce6-06d8e9f4d350"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.863411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863383 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863426 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863501 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863530 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863554 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863638 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863712 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.863977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863827 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.863977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.863840 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.864127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.864030 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.864127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.864053 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:02.864127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.864066 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.864289 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.864175 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token\") pod \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\" (UID: \"804d17fd-e5f6-4ffa-b3b2-7d044356142b\") " Oct 30 21:46:02.864338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.864287 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.865246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865215 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzlg\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-kube-api-access-ggzlg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865250 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8681933e-71c3-44b2-ba83-71d851812d50-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865266 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865280 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptq9g\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-kube-api-access-ptq9g\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865294 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865308 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8681933e-71c3-44b2-ba83-71d851812d50-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865322 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865335 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865349 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865361 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.865377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865374 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865387 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8681933e-71c3-44b2-ba83-71d851812d50-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865403 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865416 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0900c473-515d-44c1-8ce6-06d8e9f4d350-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865430 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865444 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865458 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0900c473-515d-44c1-8ce6-06d8e9f4d350-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.865472 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8681933e-71c3-44b2-ba83-71d851812d50-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:02.866413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.866028 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:02.866413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.866152 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "kube-api-access-pqwbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:02.866485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.866463 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:02.866519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:02.866487 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token" pod "804d17fd-e5f6-4ffa-b3b2-7d044356142b" (UID: "804d17fd-e5f6-4ffa-b3b2-7d044356142b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:03.033992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.016153 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:03.033992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.016191 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:03.033992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.016207 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwbs\" (UniqueName: \"kubernetes.io/projected/804d17fd-e5f6-4ffa-b3b2-7d044356142b-kube-api-access-pqwbs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:03.033992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.016223 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/804d17fd-e5f6-4ffa-b3b2-7d044356142b-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:03.332567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.332472 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" Oct 30 21:46:03.332567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.332492 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc" event={"ID":"502d78a3-3a23-4a04-87f0-89aa189dd010","Type":"ContainerDied","Data":"a64abb0311eac56937b518ecdcc0847b5802fe679547a4d97333368891418d52"} Oct 30 21:46:03.332781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.332574 5914 scope.go:122] "RemoveContainer" containerID="01f7786027aec1d1333f97ea40671b4d741c808ba713583937a2465675510c60" Oct 30 21:46:03.334019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.333864 5914 generic.go:356] "Generic (PLEG): container finished" podID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerID="429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8" exitCode=0 Oct 30 21:46:03.334019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.333917 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" event={"ID":"804d17fd-e5f6-4ffa-b3b2-7d044356142b","Type":"ContainerDied","Data":"429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8"} Oct 30 21:46:03.334019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.333961 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" event={"ID":"804d17fd-e5f6-4ffa-b3b2-7d044356142b","Type":"ContainerDied","Data":"4383f4396e71219dd3cc50cc096ae241f605f2b145efe48b139e4d6ed817efdc"} Oct 30 21:46:03.334019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.333964 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb" Oct 30 21:46:03.335640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.335622 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" Oct 30 21:46:03.335772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.335648 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7" event={"ID":"0900c473-515d-44c1-8ce6-06d8e9f4d350","Type":"ContainerDied","Data":"2eb5f510757a8c30423cfbbce2e2777e9a93cfadd8a5d4f60858bf79f70edd72"} Oct 30 21:46:03.337080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.337055 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" event={"ID":"8681933e-71c3-44b2-ba83-71d851812d50","Type":"ContainerDied","Data":"3b40f0dab9991b71129701d6a67bf3e1100759efc0c716b854f35ed0d2f79fc6"} Oct 30 21:46:03.337197 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.337111 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x" Oct 30 21:46:03.349379 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.349359 5914 scope.go:122] "RemoveContainer" containerID="429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8" Oct 30 21:46:03.359511 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.359494 5914 scope.go:122] "RemoveContainer" containerID="429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8" Oct 30 21:46:03.359806 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:03.359786 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8\": container with ID starting with 429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8 not found: ID does not exist" containerID="429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8" Oct 30 21:46:03.359911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.359813 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8"} err="failed to get container status \"429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8\": rpc error: code = NotFound desc = could not find container \"429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8\": container with ID starting with 429a3270bcf94d10cbe542a60162e63dd6b3916ce30ebfa2e579ac2d35ae1db8 not found: ID does not exist" Oct 30 21:46:03.359911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.359834 5914 scope.go:122] "RemoveContainer" containerID="e83b3d0ea486304f6787b4dbafec18c16b48c7bba7331497992db48bfe6363a3" Oct 30 21:46:03.373061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.373036 5914 scope.go:122] "RemoveContainer" containerID="e1084a421bde95c150fb3e2b2b22a87cbf677144b712b2f0f445bf6f9de536ca" Oct 30 21:46:03.503746 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.503708 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc"] Oct 30 21:46:03.508697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.508663 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-rg7dc"] Oct 30 21:46:03.577243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.577208 5914 patch_prober.go:29] interesting pod/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:46:03.577638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.577264 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:46:03.613860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.613776 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7"] Oct 30 21:46:03.619241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.619211 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-wtpr7"] Oct 30 21:46:03.728213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.728178 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x"] Oct 30 21:46:03.731584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.731553 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-k2l6x"] Oct 30 21:46:03.840184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.840151 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb"] Oct 30 21:46:03.844074 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:03.844043 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-07fa8e-openshift-default-756d98f699-8snjb"] Oct 30 21:46:04.218692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:04.218658 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" path="/var/lib/kubelet/pods/0900c473-515d-44c1-8ce6-06d8e9f4d350/volumes" Oct 30 21:46:04.219291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:04.219275 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" path="/var/lib/kubelet/pods/502d78a3-3a23-4a04-87f0-89aa189dd010/volumes" Oct 30 21:46:04.219762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:04.219748 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" path="/var/lib/kubelet/pods/804d17fd-e5f6-4ffa-b3b2-7d044356142b/volumes" Oct 30 21:46:04.220583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:04.220567 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8681933e-71c3-44b2-ba83-71d851812d50" path="/var/lib/kubelet/pods/8681933e-71c3-44b2-ba83-71d851812d50/volumes" Oct 30 21:46:04.626964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:04.625389 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x"] Oct 30 21:46:04.626964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:04.625661 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" containerID="cri-o://469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039" gracePeriod=30 Oct 30 21:46:06.276965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.276932 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7"] Oct 30 21:46:06.277370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.277132 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" podUID="da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" containerName="maas-api" containerID="cri-o://78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc" gracePeriod=30 Oct 30 21:46:06.688412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.688388 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:46:06.711555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.711528 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs\") pod \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " Oct 30 21:46:06.711695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.711627 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca\") pod \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " Oct 30 21:46:06.711695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.711679 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q\" (UniqueName: \"kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q\") pod \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " Oct 30 21:46:06.711780 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.711760 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls\") pod \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\" (UID: \"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6\") " Oct 30 21:46:06.712099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.712072 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca" pod "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" (UID: "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:06.712483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.712464 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:06.713958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.713850 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs" pod "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" (UID: "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:06.713958 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.713924 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls" pod "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" (UID: "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:06.714099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.714001 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q" pod "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" (UID: "da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6"). InnerVolumeSpecName "kube-api-access-zss9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:06.827907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.825795 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:06.827907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.825832 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zss9q\" (UniqueName: \"kubernetes.io/projected/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-kube-api-access-zss9q\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:06.827907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:06.825847 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:07.418906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.418582 5914 generic.go:356] "Generic (PLEG): container finished" podID="da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" containerID="78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc" exitCode=0 Oct 30 21:46:07.418906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.418809 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" event={"ID":"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6","Type":"ContainerDied","Data":"78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc"} Oct 30 21:46:07.418906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.418860 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" event={"ID":"da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6","Type":"ContainerDied","Data":"f7282428a156e39faf17879f30f0103e852c93252a707e5deed4293e2ff010e4"} Oct 30 21:46:07.419503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.418971 5914 scope.go:122] "RemoveContainer" containerID="78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc" Oct 30 21:46:07.419503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.419134 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7" Oct 30 21:46:07.572715 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.571866 5914 scope.go:122] "RemoveContainer" containerID="78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc" Oct 30 21:46:07.583569 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:07.583367 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc\": container with ID starting with 78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc not found: ID does not exist" containerID="78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc" Oct 30 21:46:07.583569 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.583417 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc"} err="failed to get container status \"78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc\": rpc error: code = NotFound desc = could not find container \"78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc\": container with ID starting with 78a4f37d364438ffb9afd84cd825560c41ed2650afd2b5e2c6b6754729967efc not found: ID does not exist" Oct 30 21:46:07.649972 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.649932 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7"] Oct 30 21:46:07.703926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:07.703228 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1915737c-757ff464cc-887p7"] Oct 30 21:46:08.033998 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.028286 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:46:08.074251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074223 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074281 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074314 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074357 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074509 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074578 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074636 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074608 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.074636 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074629 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.075033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074723 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:08.075033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074736 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy\") pod \"1e095191-51ca-4537-8220-6bc220e6abce\" (UID: \"1e095191-51ca-4537-8220-6bc220e6abce\") " Oct 30 21:46:08.075033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074793 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:08.075033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.074992 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:08.075290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.075089 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:08.075290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.075202 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:08.076268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.076244 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.076382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.076271 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.076382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.076285 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.076382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.076299 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.076382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.076312 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e095191-51ca-4537-8220-6bc220e6abce-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.077029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.077006 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:08.077313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.077292 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:08.077398 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.077379 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:08.077467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.077451 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx" pod "1e095191-51ca-4537-8220-6bc220e6abce" (UID: "1e095191-51ca-4537-8220-6bc220e6abce"). InnerVolumeSpecName "kube-api-access-tghsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:08.178194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.178159 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e095191-51ca-4537-8220-6bc220e6abce-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.178194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.178187 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.178194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.178197 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e095191-51ca-4537-8220-6bc220e6abce-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.178417 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.178206 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghsx\" (UniqueName: \"kubernetes.io/projected/1e095191-51ca-4537-8220-6bc220e6abce-kube-api-access-tghsx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:08.220791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.220755 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" path="/var/lib/kubelet/pods/da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6/volumes" Oct 30 21:46:08.500967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.493919 5914 generic.go:356] "Generic (PLEG): container finished" podID="1e095191-51ca-4537-8220-6bc220e6abce" containerID="a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f" exitCode=0 Oct 30 21:46:08.500967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.494048 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" event={"ID":"1e095191-51ca-4537-8220-6bc220e6abce","Type":"ContainerDied","Data":"a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f"} Oct 30 21:46:08.500967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.494103 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" event={"ID":"1e095191-51ca-4537-8220-6bc220e6abce","Type":"ContainerDied","Data":"d3b6ce8057e4f44068c4347f83bc29fdf9b73570c8cb2549d8f3e0300e971866"} Oct 30 21:46:08.500967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.494133 5914 scope.go:122] "RemoveContainer" containerID="a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f" Oct 30 21:46:08.500967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.494245 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95" Oct 30 21:46:08.536370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.534756 5914 scope.go:122] "RemoveContainer" containerID="4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993" Oct 30 21:46:08.735289 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.735252 5914 scope.go:122] "RemoveContainer" containerID="a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f" Oct 30 21:46:08.736644 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:08.736444 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f\": container with ID starting with a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f not found: ID does not exist" containerID="a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f" Oct 30 21:46:08.736644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.736485 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f"} err="failed to get container status \"a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f\": rpc error: code = NotFound desc = could not find container \"a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f\": container with ID starting with a93ce378187c79c16e201c846ece6f842068b7ab7bd6f9f63cac7258fe27f37f not found: ID does not exist" Oct 30 21:46:08.736644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.736510 5914 scope.go:122] "RemoveContainer" containerID="4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993" Oct 30 21:46:08.736644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.736597 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95"] Oct 30 21:46:08.738217 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:08.738111 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993\": container with ID starting with 4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993 not found: ID does not exist" containerID="4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993" Oct 30 21:46:08.738217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.738154 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993"} err="failed to get container status \"4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993\": rpc error: code = NotFound desc = could not find container \"4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993\": container with ID starting with 4af27041cad425c8f6f51e0f91c84688c04b78dc38659b89958514a2d7f30993 not found: ID does not exist" Oct 30 21:46:08.746444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:08.746420 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9173600a-openshift-default-5668bd4ccf-96z95"] Oct 30 21:46:09.139657 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:09.139626 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8"] Oct 30 21:46:09.140043 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:09.140008 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" containerID="cri-o://0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8" gracePeriod=30 Oct 30 21:46:09.911149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:09.911072 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:46:10.037642 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.037610 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.037833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.037697 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.037833 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.037770 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.037994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.037831 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.037994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.037947 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.037994 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.037939 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.038138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.038013 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.038138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.038052 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.038138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.038113 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.038325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.038192 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.038325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.038237 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:10.038325 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.038247 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo\") pod \"31752b73-efb9-4aa6-972b-544e87208bd9\" (UID: \"31752b73-efb9-4aa6-972b-544e87208bd9\") " Oct 30 21:46:10.039095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.039065 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.039219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.039099 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.039219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.039113 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31752b73-efb9-4aa6-972b-544e87208bd9-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.039402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.039364 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.039530 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.039509 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.041290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.041267 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "kube-api-access-89wl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.041399 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.041348 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:10.041736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.041712 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:10.041736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.041724 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token" pod "31752b73-efb9-4aa6-972b-544e87208bd9" (UID: "31752b73-efb9-4aa6-972b-544e87208bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:10.141367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.141327 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.141367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.141370 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89wl8\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-kube-api-access-89wl8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.141590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.141386 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.141590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.141401 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31752b73-efb9-4aa6-972b-544e87208bd9-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.141590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.141416 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31752b73-efb9-4aa6-972b-544e87208bd9-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.141590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.141430 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31752b73-efb9-4aa6-972b-544e87208bd9-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:10.219387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.219302 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e095191-51ca-4537-8220-6bc220e6abce" path="/var/lib/kubelet/pods/1e095191-51ca-4537-8220-6bc220e6abce/volumes" Oct 30 21:46:10.566966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.566517 5914 generic.go:356] "Generic (PLEG): container finished" podID="31752b73-efb9-4aa6-972b-544e87208bd9" containerID="469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039" exitCode=0 Oct 30 21:46:10.566966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.566591 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" Oct 30 21:46:10.566966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.566656 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" event={"ID":"31752b73-efb9-4aa6-972b-544e87208bd9","Type":"ContainerDied","Data":"469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039"} Oct 30 21:46:10.566966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.566709 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x" event={"ID":"31752b73-efb9-4aa6-972b-544e87208bd9","Type":"ContainerDied","Data":"26afec28683a9552efe966bf870a02d3f1b5b747c9cabb223105932ef17a4c25"} Oct 30 21:46:10.566966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.566740 5914 scope.go:122] "RemoveContainer" containerID="469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039" Oct 30 21:46:10.578118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.578057 5914 scope.go:122] "RemoveContainer" containerID="469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039" Oct 30 21:46:10.578407 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:10.578383 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039\": container with ID starting with 469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039 not found: ID does not exist" containerID="469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039" Oct 30 21:46:10.578515 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.578416 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039"} err="failed to get container status \"469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039\": rpc error: code = NotFound desc = could not find container \"469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039\": container with ID starting with 469043655dff6642902cad773205b1224e077ae5a88144223664e81563e17039 not found: ID does not exist" Oct 30 21:46:10.795057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.795014 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x"] Oct 30 21:46:10.836625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:10.836543 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-p6b2x"] Oct 30 21:46:12.218506 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:12.218473 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" path="/var/lib/kubelet/pods/31752b73-efb9-4aa6-972b-544e87208bd9/volumes" Oct 30 21:46:12.346446 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:12.346411 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:46:14.421137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.421075 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:46:14.521965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.521932 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.521977 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522113 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522181 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522212 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522286 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522309 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522351 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522411 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.522442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522425 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data\") pod \"cd3024ff-2a7e-4083-8d14-307097d907fd\" (UID: \"cd3024ff-2a7e-4083-8d14-307097d907fd\") " Oct 30 21:46:14.522442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522335 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:14.522715 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522693 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.522914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522831 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.523329 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.522607 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.523495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.523470 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.523581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.523505 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.523581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.523520 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.523581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.523533 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.523581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.523546 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd3024ff-2a7e-4083-8d14-307097d907fd-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.524702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.524666 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:14.524772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.524704 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:14.525243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.525216 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:14.525243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.525237 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm" pod "cd3024ff-2a7e-4083-8d14-307097d907fd" (UID: "cd3024ff-2a7e-4083-8d14-307097d907fd"). InnerVolumeSpecName "kube-api-access-tnqvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:14.600961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.600929 5914 generic.go:356] "Generic (PLEG): container finished" podID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerID="0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8" exitCode=0 Oct 30 21:46:14.601118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.600995 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" Oct 30 21:46:14.601118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.600996 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" event={"ID":"cd3024ff-2a7e-4083-8d14-307097d907fd","Type":"ContainerDied","Data":"0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8"} Oct 30 21:46:14.601118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.601060 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8" event={"ID":"cd3024ff-2a7e-4083-8d14-307097d907fd","Type":"ContainerDied","Data":"d0b2e0d616babab5e589ec2f4954fd59a9b535788666f1359dffcfc14a5a0b99"} Oct 30 21:46:14.601118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.601090 5914 scope.go:122] "RemoveContainer" containerID="0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8" Oct 30 21:46:14.614019 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.613997 5914 scope.go:122] "RemoveContainer" containerID="b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8" Oct 30 21:46:14.625275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625252 5914 scope.go:122] "RemoveContainer" containerID="0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8" Oct 30 21:46:14.625397 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625376 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.625461 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625402 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnqvm\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-kube-api-access-tnqvm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.625461 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625412 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.625461 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625428 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd3024ff-2a7e-4083-8d14-307097d907fd-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:14.625666 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:14.625646 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8\": container with ID starting with 0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8 not found: ID does not exist" containerID="0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8" Oct 30 21:46:14.625732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625693 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8"} err="failed to get container status \"0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8\": rpc error: code = NotFound desc = could not find container \"0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8\": container with ID starting with 0cc1cda6c47d97581c7c234feb41419a37f52e8db6c2afcacdafeb0dc0e4ccb8 not found: ID does not exist" Oct 30 21:46:14.625732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.625719 5914 scope.go:122] "RemoveContainer" containerID="b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8" Oct 30 21:46:14.626337 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:14.626315 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8\": container with ID starting with b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8 not found: ID does not exist" containerID="b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8" Oct 30 21:46:14.626418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.626343 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8"} err="failed to get container status \"b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8\": rpc error: code = NotFound desc = could not find container \"b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8\": container with ID starting with b19c1bb4b57487506d14917870cd58950e39324cb41da4667f4c193132ed46c8 not found: ID does not exist" Oct 30 21:46:14.750424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.750340 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8"] Oct 30 21:46:14.756974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:14.756942 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1915737c-openshift-default-5f568c4bf7-69xc8"] Oct 30 21:46:16.260929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:16.251764 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" path="/var/lib/kubelet/pods/cd3024ff-2a7e-4083-8d14-307097d907fd/volumes" Oct 30 21:46:19.743254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:19.743215 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg"] Oct 30 21:46:19.744169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:19.744132 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" containerID="cri-o://dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f" gracePeriod=30 Oct 30 21:46:19.747813 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:19.747788 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f"] Oct 30 21:46:19.748072 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:19.748041 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" containerID="cri-o://9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a" gracePeriod=30 Oct 30 21:46:24.475346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.475309 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q"] Oct 30 21:46:24.475992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.475976 5914 cpu_manager.go:414] "containerMap: removing container" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" Oct 30 21:46:24.476036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.475993 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" Oct 30 21:46:24.476036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476017 5914 cpu_manager.go:414] "containerMap: removing container" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" Oct 30 21:46:24.476036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476022 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" Oct 30 21:46:24.476036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476028 5914 cpu_manager.go:414] "containerMap: removing container" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" Oct 30 21:46:24.476036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476033 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476040 5914 cpu_manager.go:414] "containerMap: removing container" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476045 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476061 5914 cpu_manager.go:414] "containerMap: removing container" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476065 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476081 5914 cpu_manager.go:414] "containerMap: removing container" podUID="da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" containerName="maas-api" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476085 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" containerName="maas-api" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476093 5914 cpu_manager.go:414] "containerMap: removing container" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476097 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476103 5914 cpu_manager.go:414] "containerMap: removing container" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476107 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476118 5914 cpu_manager.go:414] "containerMap: removing container" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476122 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476133 5914 cpu_manager.go:414] "containerMap: removing container" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" Oct 30 21:46:24.476188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476137 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476223 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476235 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8681933e-71c3-44b2-ba83-71d851812d50" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476245 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5f0ea6-b5da-4349-b24f-f137c82de829" containerName="cleanup" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476251 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0900c473-515d-44c1-8ce6-06d8e9f4d350" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476258 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="502d78a3-3a23-4a04-87f0-89aa189dd010" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476265 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="804d17fd-e5f6-4ffa-b3b2-7d044356142b" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476273 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="da59ea7f-dd01-46d5-8bd9-f4f4a259a2c6" containerName="maas-api" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476279 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="31752b73-efb9-4aa6-972b-544e87208bd9" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476285 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" Oct 30 21:46:24.476608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.476292 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e095191-51ca-4537-8220-6bc220e6abce" containerName="istio-proxy" Oct 30 21:46:24.480172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.480155 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:24.534595 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.534569 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-3ddec2\"" Oct 30 21:46:24.566103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.566063 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:24.566276 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.566192 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hb5\" (UniqueName: \"kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5\") pod \"payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:24.588136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.588104 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q"] Oct 30 21:46:24.668178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.668143 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:24.668387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.668266 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hb5\" (UniqueName: \"kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5\") pod \"payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:24.668921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.668850 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:24.951914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:24.948709 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hb5\" (UniqueName: \"kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5\") pod \"payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:25.095679 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.095347 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:25.213894 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.213564 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:46:25.281128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.281094 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:46:25.353583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.353491 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.353583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.353554 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.353821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.353636 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.353821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.353683 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.353821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.353735 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.354160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.353949 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.354160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354013 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.354433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354411 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.354529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354453 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.354529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354471 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:25.354529 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354516 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.354667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354561 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.354667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354600 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.354667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354648 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.354819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354693 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.354819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354721 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.354819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354774 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.355160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354828 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.355160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354946 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx\") pod \"7e661bb3-f3ee-48e5-a053-06fe0a677403\" (UID: \"7e661bb3-f3ee-48e5-a053-06fe0a677403\") " Oct 30 21:46:25.355160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.354983 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.355160 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.355061 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data\") pod \"1379ca66-23f3-43b9-b98b-786ab7d9f763\" (UID: \"1379ca66-23f3-43b9-b98b-786ab7d9f763\") " Oct 30 21:46:25.356045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.355649 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.356045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.355737 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.356045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.355922 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.356263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356101 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.356263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356128 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.356263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356142 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.356263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356156 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.356263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356170 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.356263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356185 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e661bb3-f3ee-48e5-a053-06fe0a677403-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.356584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356500 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:25.356776 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356754 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.357002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.356979 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.357184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.357161 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.362762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.362723 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.366378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.366349 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.372395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.372367 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.372395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.372375 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "kube-api-access-crrdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.372627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.372411 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo" pod "1379ca66-23f3-43b9-b98b-786ab7d9f763" (UID: "1379ca66-23f3-43b9-b98b-786ab7d9f763"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:25.372627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.372470 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:46:25.372627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.372505 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:46:25.374487 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.374450 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx" pod "7e661bb3-f3ee-48e5-a053-06fe0a677403" (UID: "7e661bb3-f3ee-48e5-a053-06fe0a677403"). InnerVolumeSpecName "kube-api-access-kx4gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:25.457831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457791 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.457831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457833 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457847 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457863 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457922 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457938 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e661bb3-f3ee-48e5-a053-06fe0a677403-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457952 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1379ca66-23f3-43b9-b98b-786ab7d9f763-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457967 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457981 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crrdq\" (UniqueName: \"kubernetes.io/projected/1379ca66-23f3-43b9-b98b-786ab7d9f763-kube-api-access-crrdq\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.457997 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.458011 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4gx\" (UniqueName: \"kubernetes.io/projected/7e661bb3-f3ee-48e5-a053-06fe0a677403-kube-api-access-kx4gx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.458155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.458023 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1379ca66-23f3-43b9-b98b-786ab7d9f763-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:46:25.500938 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.500900 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q"] Oct 30 21:46:25.516506 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:46:25.516474 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7c967b_3dad_4950_b77e_005dfb518f9c.slice/crio-74cf7236aea4e778aa49b1e372d12b863069d8b333e13d60200bd4f893305237 WatchSource:0}: Error finding container 74cf7236aea4e778aa49b1e372d12b863069d8b333e13d60200bd4f893305237: Status 404 returned error can't find the container with id 74cf7236aea4e778aa49b1e372d12b863069d8b333e13d60200bd4f893305237 Oct 30 21:46:25.844022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.843986 5914 generic.go:356] "Generic (PLEG): container finished" podID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerID="dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f" exitCode=0 Oct 30 21:46:25.844314 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.844081 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" event={"ID":"1379ca66-23f3-43b9-b98b-786ab7d9f763","Type":"ContainerDied","Data":"dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f"} Oct 30 21:46:25.844314 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.844123 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" event={"ID":"1379ca66-23f3-43b9-b98b-786ab7d9f763","Type":"ContainerDied","Data":"a19665f9ecd6a5b43ce564129445fed7ec85fec759da4d4cf165c2b4c0832405"} Oct 30 21:46:25.844314 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.844149 5914 scope.go:122] "RemoveContainer" containerID="dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f" Oct 30 21:46:25.844314 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.844263 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg" Oct 30 21:46:25.847382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.847300 5914 generic.go:356] "Generic (PLEG): container finished" podID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerID="9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a" exitCode=0 Oct 30 21:46:25.847516 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.847392 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" Oct 30 21:46:25.847516 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.847429 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" event={"ID":"7e661bb3-f3ee-48e5-a053-06fe0a677403","Type":"ContainerDied","Data":"9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a"} Oct 30 21:46:25.847516 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.847479 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f" event={"ID":"7e661bb3-f3ee-48e5-a053-06fe0a677403","Type":"ContainerDied","Data":"6dd532dbbad9b85b24bc994b99a26bcba8e9c45eb3658a31c12e4f85037c2ac6"} Oct 30 21:46:25.856346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.855935 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" event={"ID":"6f7c967b-3dad-4950-b77e-005dfb518f9c","Type":"ContainerStarted","Data":"462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f"} Oct 30 21:46:25.856346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.856022 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" event={"ID":"6f7c967b-3dad-4950-b77e-005dfb518f9c","Type":"ContainerStarted","Data":"74cf7236aea4e778aa49b1e372d12b863069d8b333e13d60200bd4f893305237"} Oct 30 21:46:25.856346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.856084 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:25.858714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.858529 5914 scope.go:122] "RemoveContainer" containerID="dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f" Oct 30 21:46:25.858977 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:25.858859 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f\": container with ID starting with dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f not found: ID does not exist" containerID="dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f" Oct 30 21:46:25.858977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.858918 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f"} err="failed to get container status \"dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f\": rpc error: code = NotFound desc = could not find container \"dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f\": container with ID starting with dfa4d474d59285c121be0a670ff6478dab65f7401574913ed118baf885213d0f not found: ID does not exist" Oct 30 21:46:25.858977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.858942 5914 scope.go:122] "RemoveContainer" containerID="9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a" Oct 30 21:46:25.869226 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.869207 5914 scope.go:122] "RemoveContainer" containerID="9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a" Oct 30 21:46:25.869538 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:25.869516 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a\": container with ID starting with 9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a not found: ID does not exist" containerID="9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a" Oct 30 21:46:25.869588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:25.869548 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a"} err="failed to get container status \"9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a\": rpc error: code = NotFound desc = could not find container \"9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a\": container with ID starting with 9ca5960b217975b171b9230559a29db086d9e4e932b1e700cc0a410ec500221a not found: ID does not exist" Oct 30 21:46:26.011022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:26.010927 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg"] Oct 30 21:46:26.014696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:26.014673 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-jvngg"] Oct 30 21:46:26.128048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:26.127982 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" podStartSLOduration=2.127962004 podStartE2EDuration="2.127962004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:26.123745898 +0000 UTC m=+1738.025609962" watchObservedRunningTime="2026-10-30 21:46:26.127962004 +0000 UTC m=+1738.029826069" Oct 30 21:46:26.221076 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:26.221039 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" path="/var/lib/kubelet/pods/1379ca66-23f3-43b9-b98b-786ab7d9f763/volumes" Oct 30 21:46:26.281828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:26.273937 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f"] Oct 30 21:46:26.289913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:26.288538 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-w8b4f"] Oct 30 21:46:28.219135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:28.219103 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" path="/var/lib/kubelet/pods/7e661bb3-f3ee-48e5-a053-06fe0a677403/volumes" Oct 30 21:46:28.559609 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:28.559520 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:46:36.893809 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:46:36.893778 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:46:42.347974 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:46:42.347927 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:47:04.417053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417021 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4"] Oct 30 21:47:04.417687 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417672 5914 cpu_manager.go:414] "containerMap: removing container" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" Oct 30 21:47:04.417745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417688 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" Oct 30 21:47:04.417745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417701 5914 cpu_manager.go:414] "containerMap: removing container" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" Oct 30 21:47:04.417745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417708 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" Oct 30 21:47:04.417847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417838 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e661bb3-f3ee-48e5-a053-06fe0a677403" containerName="istio-proxy" Oct 30 21:47:04.417898 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417850 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1379ca66-23f3-43b9-b98b-786ab7d9f763" containerName="istio-proxy" Oct 30 21:47:04.417898 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.417858 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3024ff-2a7e-4083-8d14-307097d907fd" containerName="istio-proxy" Oct 30 21:47:04.421628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.421612 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.478992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.478961 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4"] Oct 30 21:47:04.508679 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.508641 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.508847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.508817 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.508935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.508903 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.508981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.508950 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8d5\" (UniqueName: \"kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.528029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.528005 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-218800-serving-cert\"" Oct 30 21:47:04.610655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.610623 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.610829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.610743 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.610829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.610804 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.610923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.610852 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8d5\" (UniqueName: \"kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.610970 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:47:04.610927 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-serving-cert: secret "maas-api-e2e-shared-a-w4-218800-serving-cert" not found Oct 30 21:47:04.611014 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:47:04.611001 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls podName:1d81c1ad-ba94-4417-a066-ae7a1e74c2b9 nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.11097995 +0000 UTC m=+1777.012844006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls") pod "maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" (UID: "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9") : secret "maas-api-e2e-shared-a-w4-218800-serving-cert" not found Oct 30 21:47:04.611346 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.611327 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.613383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.613364 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:04.713964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:04.699493 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8d5\" (UniqueName: \"kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:05.124190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:05.124155 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:05.126822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:05.126794 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:05.337038 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:05.337006 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:05.723967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:05.723934 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4"] Oct 30 21:47:05.725007 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:47:05.724972 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d81c1ad_ba94_4417_a066_ae7a1e74c2b9.slice/crio-22c0ee4c33aea4d770cd3c276cf6508bfc87ea01d711e098e10c956066e657df WatchSource:0}: Error finding container 22c0ee4c33aea4d770cd3c276cf6508bfc87ea01d711e098e10c956066e657df: Status 404 returned error can't find the container with id 22c0ee4c33aea4d770cd3c276cf6508bfc87ea01d711e098e10c956066e657df Oct 30 21:47:05.726490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:05.726471 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:47:06.294294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:06.294257 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" event={"ID":"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9","Type":"ContainerStarted","Data":"22c0ee4c33aea4d770cd3c276cf6508bfc87ea01d711e098e10c956066e657df"} Oct 30 21:47:07.302683 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:07.302644 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" event={"ID":"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9","Type":"ContainerStarted","Data":"688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8"} Oct 30 21:47:07.303121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:07.302721 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:12.348414 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:47:12.348367 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:47:13.315443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:13.315411 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:47:13.433691 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:13.433633 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" podStartSLOduration=8.574216308 podStartE2EDuration="9.433619847s" totalImagesPullingTime="859.403539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:07.434547715 +0000 UTC m=+1779.336411780" watchObservedRunningTime="2026-10-30 21:47:13.433619847 +0000 UTC m=+1785.335483911" Oct 30 21:47:28.510190 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.510158 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5"] Oct 30 21:47:28.514548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.514527 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.567491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.567454 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-3ddec2-serving-cert\"" Oct 30 21:47:28.589342 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.589300 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.589551 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.589435 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.589551 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.589527 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.589689 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.589649 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87d7z\" (UniqueName: \"kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.619799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.619768 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5"] Oct 30 21:47:28.691004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.690969 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.691194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.691029 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.691194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.691075 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.691194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.691175 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87d7z\" (UniqueName: \"kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.691382 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:47:28.691241 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-serving-cert: secret "maas-api-e2e-shared-b-w1-3ddec2-serving-cert" not found Oct 30 21:47:28.691382 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:47:28.691313 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls podName:6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c nodeName:}" failed. No retries permitted until 2026-10-30 21:47:29.191292339 +0000 UTC m=+1801.093156399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls") pod "maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" (UID: "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c") : secret "maas-api-e2e-shared-b-w1-3ddec2-serving-cert" not found Oct 30 21:47:28.691800 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.691775 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.693925 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.693901 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:28.750264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:28.750234 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87d7z\" (UniqueName: \"kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:29.123860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:29.123825 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:47:29.124268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:29.124244 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:47:29.227906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:29.227847 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:29.230482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:29.230457 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:29.429757 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:29.429664 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:29.812625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:29.812594 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5"] Oct 30 21:47:29.813703 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:47:29.813674 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9d8b7b_d3b5_467d_a1d7_18410ef5e47c.slice/crio-9acf5f1fd4fae8cba3ebaf1e3479ed990a94c18b5af016fe1b72a5dac3f5b22e WatchSource:0}: Error finding container 9acf5f1fd4fae8cba3ebaf1e3479ed990a94c18b5af016fe1b72a5dac3f5b22e: Status 404 returned error can't find the container with id 9acf5f1fd4fae8cba3ebaf1e3479ed990a94c18b5af016fe1b72a5dac3f5b22e Oct 30 21:47:30.507935 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:30.507866 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" event={"ID":"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c","Type":"ContainerStarted","Data":"9acf5f1fd4fae8cba3ebaf1e3479ed990a94c18b5af016fe1b72a5dac3f5b22e"} Oct 30 21:47:31.517764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:31.517726 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" event={"ID":"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c","Type":"ContainerStarted","Data":"ed826b12af4c82de5282cb445da822885c455e0f6af4a1fc6d95fd4fe2be87ff"} Oct 30 21:47:31.518317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:31.517924 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:31.639976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:31.639921 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" podStartSLOduration=2.867300653 podStartE2EDuration="3.639905299s" totalImagesPullingTime="772.604646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:31.638151125 +0000 UTC m=+1803.540015191" watchObservedRunningTime="2026-10-30 21:47:31.639905299 +0000 UTC m=+1803.541769363" Oct 30 21:47:33.025471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:33.025428 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:47:37.661974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:37.658438 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:47:40.794460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.794417 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4"] Oct 30 21:47:40.799436 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.799414 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824341 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824307 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824352 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824455 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824502 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824535 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824670 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmdml\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.824865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.824842 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.825022 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.825005 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.825085 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.825054 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.851873 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.851848 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-218800-openshift-default-dockercfg-696vk\"" Oct 30 21:47:40.906971 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.906934 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4"] Oct 30 21:47:40.926952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.926907 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.926952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.926948 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927015 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmdml\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927142 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927262 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927232 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927279 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927331 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927356 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927396 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927413 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927608 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927578 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.927900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.927852 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.928038 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.928014 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.928163 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.928143 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.930676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.929665 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.930676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.929989 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.984736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.984700 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:40.984896 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:40.984749 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmdml\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml\") pod \"e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:41.114855 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.114763 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:41.546758 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.546728 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4"] Oct 30 21:47:41.547558 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:47:41.547528 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58507410_d70e_4a94_8e65_9861b2d7bf99.slice/crio-935b950c507ed7f2680ea1f256aa8574cb9be27c1bfee8b831f394f1946aa86b WatchSource:0}: Error finding container 935b950c507ed7f2680ea1f256aa8574cb9be27c1bfee8b831f394f1946aa86b: Status 404 returned error can't find the container with id 935b950c507ed7f2680ea1f256aa8574cb9be27c1bfee8b831f394f1946aa86b Oct 30 21:47:41.550372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.550338 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:41.550458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.550405 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:41.550458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.550442 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:47:41.682957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.682914 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" event={"ID":"58507410-d70e-4a94-8e65-9861b2d7bf99","Type":"ContainerStarted","Data":"3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706"} Oct 30 21:47:41.683166 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.682981 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" event={"ID":"58507410-d70e-4a94-8e65-9861b2d7bf99","Type":"ContainerStarted","Data":"935b950c507ed7f2680ea1f256aa8574cb9be27c1bfee8b831f394f1946aa86b"} Oct 30 21:47:41.802481 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:41.802365 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podStartSLOduration=1.802346187 podStartE2EDuration="1.802346187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:41.801351758 +0000 UTC m=+1813.703215835" watchObservedRunningTime="2026-10-30 21:47:41.802346187 +0000 UTC m=+1813.704210253" Oct 30 21:47:42.115321 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:42.115234 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:42.116554 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:42.116529 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:47:42.116606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:42.116579 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 21:47:42.346707 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:47:42.346670 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:47:43.116227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:43.116187 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:47:43.116643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:43.116263 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 21:47:44.115696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:44.115660 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:47:44.116006 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:44.115729 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 21:47:45.115214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:45.115178 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:47:45.115631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:45.115230 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 21:47:46.115607 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:46.115566 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:47:46.116070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:46.115635 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 21:47:47.115904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:47.115841 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 21:47:47.116317 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:47.115942 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 21:47:48.120365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:48.120335 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:48.120817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:48.120676 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:47:48.121280 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:47:48.121263 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:48:00.373847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:00.373802 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:00.393208 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:00.393165 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:00.463034 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:00.462975 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:00.527862 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:00.527815 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 21:48:00.533991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:00.533948 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:00.697108 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:00.696998 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:12.346808 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:48:12.346761 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:48:12.941919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:12.941870 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl"] Oct 30 21:48:12.942119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:12.942096 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" containerID="cri-o://5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091" gracePeriod=30 Oct 30 21:48:17.379710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.379666 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj"] Oct 30 21:48:17.384324 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.384302 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.436207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.436170 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-3ddec2-openshift-default-dockercfg-bvfmv\"" Oct 30 21:48:17.501003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.500968 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501225 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501331 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501281 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501371 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501353 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501394 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmc8\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501448 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501435 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501600 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501583 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501644 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501620 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.501692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.501678 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.588219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.588186 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj"] Oct 30 21:48:17.603168 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.603132 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.603442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.603415 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.603643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.603623 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.603817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.603798 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604106 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604149 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.603840 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604247 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604431 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604295 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmc8\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604431 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604339 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604516 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604535 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.604807 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.604781 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.605255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.605227 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.607713 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.607691 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.607815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.607793 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.661053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.660981 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.662408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.662379 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmc8\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:17.701747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:17.701704 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:18.151928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.151855 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj"] Oct 30 21:48:18.155017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.154983 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.155107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.155051 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.155107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.155082 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:18.202362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.202302 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:48:18.211652 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.211624 5914 generic.go:356] "Generic (PLEG): container finished" podID="bcda770e-128b-454c-a843-38916d2f7603" containerID="5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091" exitCode=0 Oct 30 21:48:18.211792 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.211730 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" Oct 30 21:48:18.224247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.224212 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" event={"ID":"bcda770e-128b-454c-a843-38916d2f7603","Type":"ContainerDied","Data":"5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091"} Oct 30 21:48:18.224424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.224280 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl" event={"ID":"bcda770e-128b-454c-a843-38916d2f7603","Type":"ContainerDied","Data":"55cff74f6aa03f9d4285d0775649ff64829a15d15b275ebbc3cc5b0fd90a2415"} Oct 30 21:48:18.224424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.224313 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" event={"ID":"0b0f307a-eec1-49f8-86b3-8cb9220c6313","Type":"ContainerStarted","Data":"d6cdaa84a47dcf6ae2c9063cb32b9251823b2bc5d470a8bf49b7b488a10252b5"} Oct 30 21:48:18.224424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.224358 5914 scope.go:122] "RemoveContainer" containerID="5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091" Oct 30 21:48:18.236826 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.236802 5914 scope.go:122] "RemoveContainer" containerID="5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091" Oct 30 21:48:18.237204 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:48:18.237171 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091\": container with ID starting with 5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091 not found: ID does not exist" containerID="5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091" Oct 30 21:48:18.237295 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.237220 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091"} err="failed to get container status \"5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091\": rpc error: code = NotFound desc = could not find container \"5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091\": container with ID starting with 5f3ccdebf9ec363ff5274501f2bfe6198255ce5f0165896c8f62233ce4664091 not found: ID does not exist" Oct 30 21:48:18.321256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321221 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321464 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321298 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321464 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321325 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321464 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321544 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321601 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321706 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321746 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token\") pod \"bcda770e-128b-454c-a843-38916d2f7603\" (UID: \"bcda770e-128b-454c-a843-38916d2f7603\") " Oct 30 21:48:18.321950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.321842 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322130 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322143 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322569 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcda770e-128b-454c-a843-38916d2f7603-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322587 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322602 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322658 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:18.322732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.322708 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:18.324992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.324947 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:18.325489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.325438 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:18.327125 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.327096 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "kube-api-access-ph7cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:18.327236 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.327209 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo" pod "bcda770e-128b-454c-a843-38916d2f7603" (UID: "bcda770e-128b-454c-a843-38916d2f7603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:18.426946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.425314 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.426946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.425395 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.426946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.425429 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7cx\" (UniqueName: \"kubernetes.io/projected/bcda770e-128b-454c-a843-38916d2f7603-kube-api-access-ph7cx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.426946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.425447 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.426946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.425477 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcda770e-128b-454c-a843-38916d2f7603-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.426946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.425494 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcda770e-128b-454c-a843-38916d2f7603-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:18.638841 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.638807 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl"] Oct 30 21:48:18.643968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:18.643944 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-zg7bl"] Oct 30 21:48:19.222327 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:19.222283 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" event={"ID":"0b0f307a-eec1-49f8-86b3-8cb9220c6313","Type":"ContainerStarted","Data":"384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9"} Oct 30 21:48:19.341802 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:19.341746 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podStartSLOduration=2.341732848 podStartE2EDuration="2.341732848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:19.340667989 +0000 UTC m=+1851.242532081" watchObservedRunningTime="2026-10-30 21:48:19.341732848 +0000 UTC m=+1851.243596914" Oct 30 21:48:19.702235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:19.702202 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:19.704217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:19.704186 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:19.704365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:19.704247 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 21:48:20.218522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:20.218488 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcda770e-128b-454c-a843-38916d2f7603" path="/var/lib/kubelet/pods/bcda770e-128b-454c-a843-38916d2f7603/volumes" Oct 30 21:48:20.647580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:20.647545 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h"] Oct 30 21:48:20.647766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:20.647744 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" containerID="cri-o://9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d" gracePeriod=30 Oct 30 21:48:20.705908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:20.702387 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:20.705908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:20.702447 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 21:48:21.702836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:21.702802 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:21.703065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:21.702866 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 21:48:22.702301 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:22.702262 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:22.702684 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:22.702316 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 21:48:23.702861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:23.702821 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 21:48:23.703351 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:23.702915 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 21:48:24.739361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:24.739326 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:24.740194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:24.740167 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:24.741492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:24.741472 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:48:24.904064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:24.904030 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh"] Oct 30 21:48:24.904294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:24.904265 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" containerID="cri-o://f304e027cad8b0dca0643be78cd94226ee3febd3c05fc29c6168f8e2772a3e49" gracePeriod=30 Oct 30 21:48:25.930113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:25.930053 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:48:26.004283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004248 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004332 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004412 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004434 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004482 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004575 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:26.004606 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004588 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:26.004692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004623 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004692 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004665 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004706 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004824 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004770 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:26.004824 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004792 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo\") pod \"89f822d9-ca18-4779-a370-a289ba734096\" (UID: \"89f822d9-ca18-4779-a370-a289ba734096\") " Oct 30 21:48:26.004977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004865 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:26.004977 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.004873 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:26.005752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.005725 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.005911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.005755 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f822d9-ca18-4779-a370-a289ba734096-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.005911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.005772 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.005911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.005787 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.005911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.005802 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.006971 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.006945 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:26.007055 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.006973 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:26.007126 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.007110 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "kube-api-access-b8znj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:26.007239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.007224 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo" pod "89f822d9-ca18-4779-a370-a289ba734096" (UID: "89f822d9-ca18-4779-a370-a289ba734096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:26.107919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.107857 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f822d9-ca18-4779-a370-a289ba734096-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.107919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.107920 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.108196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.107941 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8znj\" (UniqueName: \"kubernetes.io/projected/89f822d9-ca18-4779-a370-a289ba734096-kube-api-access-b8znj\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.108196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.107957 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f822d9-ca18-4779-a370-a289ba734096-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:26.334740 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.334703 5914 generic.go:356] "Generic (PLEG): container finished" podID="89f822d9-ca18-4779-a370-a289ba734096" containerID="9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d" exitCode=0 Oct 30 21:48:26.334940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.334751 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" Oct 30 21:48:26.334940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.334791 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" event={"ID":"89f822d9-ca18-4779-a370-a289ba734096","Type":"ContainerDied","Data":"9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d"} Oct 30 21:48:26.334940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.334848 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h" event={"ID":"89f822d9-ca18-4779-a370-a289ba734096","Type":"ContainerDied","Data":"2003cafaaa0a7aa47909e08523a690b8a84bb573b624b3d987e03257cafe3480"} Oct 30 21:48:26.334940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.334911 5914 scope.go:122] "RemoveContainer" containerID="9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d" Oct 30 21:48:26.346088 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.346058 5914 scope.go:122] "RemoveContainer" containerID="9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d" Oct 30 21:48:26.346492 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:48:26.346466 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d\": container with ID starting with 9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d not found: ID does not exist" containerID="9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d" Oct 30 21:48:26.346611 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.346501 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d"} err="failed to get container status \"9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d\": rpc error: code = NotFound desc = could not find container \"9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d\": container with ID starting with 9253dd8a15132a07b40e8fdb7a4e305517254abfb4e0e87568986a8857f32b8d not found: ID does not exist" Oct 30 21:48:26.502161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.502070 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h"] Oct 30 21:48:26.505591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:26.505563 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-hjl5h"] Oct 30 21:48:27.805905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:27.804347 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:48:28.475682 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:28.475643 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f822d9-ca18-4779-a370-a289ba734096" path="/var/lib/kubelet/pods/89f822d9-ca18-4779-a370-a289ba734096/volumes" Oct 30 21:48:29.646231 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:29.646192 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g"] Oct 30 21:48:29.647140 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:29.647104 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" podUID="6cc1d63f-f5db-4727-bee9-5bee78d38e8a" containerName="maas-api" containerID="cri-o://e6ae3607f1781f614b68984c30f25cb9ebb22cf99c28f780eab1cf3d4de32323" gracePeriod=30 Oct 30 21:48:30.676978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.671375 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:48:30.685342 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.671465 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:48:30.894951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.894921 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:48:30.895746 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.895713 5914 generic.go:356] "Generic (PLEG): container finished" podID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerID="f304e027cad8b0dca0643be78cd94226ee3febd3c05fc29c6168f8e2772a3e49" exitCode=0 Oct 30 21:48:30.896344 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.896315 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" event={"ID":"7dc5d51e-5931-4fdf-ac4f-aaff7658b730","Type":"ContainerDied","Data":"f304e027cad8b0dca0643be78cd94226ee3febd3c05fc29c6168f8e2772a3e49"} Oct 30 21:48:30.903647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.903613 5914 generic.go:356] "Generic (PLEG): container finished" podID="6cc1d63f-f5db-4727-bee9-5bee78d38e8a" containerID="e6ae3607f1781f614b68984c30f25cb9ebb22cf99c28f780eab1cf3d4de32323" exitCode=0 Oct 30 21:48:30.913975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.903870 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" event={"ID":"6cc1d63f-f5db-4727-bee9-5bee78d38e8a","Type":"ContainerDied","Data":"e6ae3607f1781f614b68984c30f25cb9ebb22cf99c28f780eab1cf3d4de32323"} Oct 30 21:48:30.914352 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.914328 5914 scope.go:122] "RemoveContainer" containerID="e6ae3607f1781f614b68984c30f25cb9ebb22cf99c28f780eab1cf3d4de32323" Oct 30 21:48:30.914931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:30.914912 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" Oct 30 21:48:31.054046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.038201 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca\") pod \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " Oct 30 21:48:31.078533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.078215 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca" pod "6cc1d63f-f5db-4727-bee9-5bee78d38e8a" (UID: "6cc1d63f-f5db-4727-bee9-5bee78d38e8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:31.183265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.183213 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs\") pod \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " Oct 30 21:48:31.183588 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.183562 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls\") pod \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " Oct 30 21:48:31.183769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.183749 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n\" (UniqueName: \"kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n\") pod \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\" (UID: \"6cc1d63f-f5db-4727-bee9-5bee78d38e8a\") " Oct 30 21:48:31.197661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.197619 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:31.471819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.278647 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:31.498591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.412336 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n" pod "6cc1d63f-f5db-4727-bee9-5bee78d38e8a" (UID: "6cc1d63f-f5db-4727-bee9-5bee78d38e8a"). InnerVolumeSpecName "kube-api-access-29x8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:31.498823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.412414 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs" pod "6cc1d63f-f5db-4727-bee9-5bee78d38e8a" (UID: "6cc1d63f-f5db-4727-bee9-5bee78d38e8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:31.498986 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.451446 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:48:31.499377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.451269 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:31.881382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:31.862230 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls" pod "6cc1d63f-f5db-4727-bee9-5bee78d38e8a" (UID: "6cc1d63f-f5db-4727-bee9-5bee78d38e8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:32.264910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.248734 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.265246 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.265216 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29x8n\" (UniqueName: \"kubernetes.io/projected/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-kube-api-access-29x8n\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.265382 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.265367 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc1d63f-f5db-4727-bee9-5bee78d38e8a-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.297623 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.297574 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:32.416941 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.413465 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:32.502267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.500006 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:32.502267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.500328 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:48:32.556965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.556919 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:48:32.580012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.579967 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.607644 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.607793 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.607842 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.608014 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.608045 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.608161 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.608238 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.608266 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.608336 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo\") pod \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\" (UID: \"7dc5d51e-5931-4fdf-ac4f-aaff7658b730\") " Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.621050 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:32.629904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.622101 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:32.639108 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.633824 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:48:32.640037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.640013 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:48:32.640235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.640212 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g" event={"ID":"6cc1d63f-f5db-4727-bee9-5bee78d38e8a","Type":"ContainerDied","Data":"ba7cbd3bc9514273c6983f324991d720339d82ac247875db9fd52446aca67f4d"} Oct 30 21:48:32.640378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.640364 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:48:32.640489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.640474 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" event={"ID":"7dc5d51e-5931-4fdf-ac4f-aaff7658b730","Type":"ContainerDied","Data":"95fa03211eab2ed1c8f7ef847fcf2b831c591a38253db04915a9d6473607b152"} Oct 30 21:48:32.642519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.642497 5914 scope.go:122] "RemoveContainer" containerID="f304e027cad8b0dca0643be78cd94226ee3febd3c05fc29c6168f8e2772a3e49" Oct 30 21:48:32.647421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.647081 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:32.647421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.647210 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:32.647597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.647555 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:32.668950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.663097 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:32.668950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.663572 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:32.670206 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.670163 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:32.677162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.673550 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5" pod "7dc5d51e-5931-4fdf-ac4f-aaff7658b730" (UID: "7dc5d51e-5931-4fdf-ac4f-aaff7658b730"). InnerVolumeSpecName "kube-api-access-w7xw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716387 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716418 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716432 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716446 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716460 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716473 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716489 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716502 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.718675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.716514 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xw5\" (UniqueName: \"kubernetes.io/projected/7dc5d51e-5931-4fdf-ac4f-aaff7658b730-kube-api-access-w7xw5\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:32.761584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:32.761552 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:48:33.017822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:33.017783 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g"] Oct 30 21:48:33.021565 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:33.021537 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-29b7ead1-69f77cfb9-62x4g"] Oct 30 21:48:33.502960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:33.502928 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh" Oct 30 21:48:33.668501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:33.668464 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh"] Oct 30 21:48:33.718423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:33.718385 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-qdxfh"] Oct 30 21:48:34.138954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.138917 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn"] Oct 30 21:48:34.139623 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139607 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6cc1d63f-f5db-4727-bee9-5bee78d38e8a" containerName="maas-api" Oct 30 21:48:34.139667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139624 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc1d63f-f5db-4727-bee9-5bee78d38e8a" containerName="maas-api" Oct 30 21:48:34.139667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139645 5914 cpu_manager.go:414] "containerMap: removing container" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" Oct 30 21:48:34.139667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139650 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" Oct 30 21:48:34.139667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139667 5914 cpu_manager.go:414] "containerMap: removing container" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" Oct 30 21:48:34.139803 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139672 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" Oct 30 21:48:34.139803 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139691 5914 cpu_manager.go:414] "containerMap: removing container" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" Oct 30 21:48:34.139803 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139700 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" Oct 30 21:48:34.140017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139817 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcda770e-128b-454c-a843-38916d2f7603" containerName="istio-proxy" Oct 30 21:48:34.140017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139833 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc1d63f-f5db-4727-bee9-5bee78d38e8a" containerName="maas-api" Oct 30 21:48:34.140017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139842 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" containerName="istio-proxy" Oct 30 21:48:34.140017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.139851 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f822d9-ca18-4779-a370-a289ba734096" containerName="istio-proxy" Oct 30 21:48:34.143498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.143479 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.200961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.200929 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4a1551-openshift-default-dockercfg-lhmp4\"" Oct 30 21:48:34.243974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.240610 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn"] Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.246815 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259158 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259204 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259251 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259280 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfj8\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259349 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259409 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259472 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.266903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.259526 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.367695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.367650 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.368061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.368032 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.378007 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.377969 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.395965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.378356 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.395965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.378396 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfj8\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.395965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.378483 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.395965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.378547 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.395965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.378616 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.422907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.397116 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.423178 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.413010 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.423397 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.422834 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.427200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.427170 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.427342 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.427293 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.427407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.427382 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.444951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.431493 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.525685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.524940 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.537919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.532373 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc1d63f-f5db-4727-bee9-5bee78d38e8a" path="/var/lib/kubelet/pods/6cc1d63f-f5db-4727-bee9-5bee78d38e8a/volumes" Oct 30 21:48:34.537919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.536495 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfj8\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.548007 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.543495 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token\") pod \"e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:34.548007 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.543578 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc5d51e-5931-4fdf-ac4f-aaff7658b730" path="/var/lib/kubelet/pods/7dc5d51e-5931-4fdf-ac4f-aaff7658b730/volumes" Oct 30 21:48:34.795425 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:34.795338 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:35.233244 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.233214 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn"] Oct 30 21:48:35.233594 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:48:35.233298 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7357f2_2289_4074_bf14_fe315ad33e0a.slice/crio-23855262569eeab91d934768394a73b817133896ed1da5c03c425eba0e8e56b8 WatchSource:0}: Error finding container 23855262569eeab91d934768394a73b817133896ed1da5c03c425eba0e8e56b8: Status 404 returned error can't find the container with id 23855262569eeab91d934768394a73b817133896ed1da5c03c425eba0e8e56b8 Oct 30 21:48:35.236092 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.236062 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:35.236169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.236126 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:35.236169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.236157 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:35.564492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.564385 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" event={"ID":"3a7357f2-2289-4074-bf14-fe315ad33e0a","Type":"ContainerStarted","Data":"447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe"} Oct 30 21:48:35.564492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.564451 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" event={"ID":"3a7357f2-2289-4074-bf14-fe315ad33e0a","Type":"ContainerStarted","Data":"23855262569eeab91d934768394a73b817133896ed1da5c03c425eba0e8e56b8"} Oct 30 21:48:35.684281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.684209 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podStartSLOduration=1.684189615 podStartE2EDuration="1.684189615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:35.67996836 +0000 UTC m=+1867.581832425" watchObservedRunningTime="2026-10-30 21:48:35.684189615 +0000 UTC m=+1867.586053681" Oct 30 21:48:35.802930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:35.797993 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:36.802523 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:36.802487 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:37.579165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:37.579130 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:37.580012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:37.579991 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:48:40.833603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.833561 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6"] Oct 30 21:48:40.838610 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.838583 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.890726 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.890691 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50b188-openshift-default-dockercfg-jb28j\"" Oct 30 21:48:40.919224 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919186 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919342 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919406 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mmr\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919454 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919521 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919554 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919728 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919664 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919728 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919688 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.919835 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.919789 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:40.944131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:40.944092 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6"] Oct 30 21:48:41.095145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095043 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095121 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095225 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095254 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095383 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095372 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095487 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095636 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095715 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095677 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mmr\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095751 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.095823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.095774 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.096228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.096200 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.096484 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.096461 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.096711 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.096691 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.097225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.097199 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.100021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.099993 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.100171 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.100041 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.153105 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.153069 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.153270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.153141 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mmr\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr\") pod \"e2e-inf-a-50b188-openshift-default-768d545959-5ksg6\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.154020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.153992 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:41.666149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.666106 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6"] Oct 30 21:48:41.676621 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:48:41.676577 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacb3b25_4d18_484d_8aba_73b61f6a411e.slice/crio-85857ac1e7541fa234f5d73910dd5b8ae4d476cd06b64b472f4c585e1d498d6a WatchSource:0}: Error finding container 85857ac1e7541fa234f5d73910dd5b8ae4d476cd06b64b472f4c585e1d498d6a: Status 404 returned error can't find the container with id 85857ac1e7541fa234f5d73910dd5b8ae4d476cd06b64b472f4c585e1d498d6a Oct 30 21:48:41.679474 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.679434 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:41.679559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.679515 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:41.679559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:41.679549 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:42.528077 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:48:42.528032 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:48:42.641298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:42.641253 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" event={"ID":"aacb3b25-4d18-484d-8aba-73b61f6a411e","Type":"ContainerStarted","Data":"a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b"} Oct 30 21:48:42.641515 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:42.641312 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" event={"ID":"aacb3b25-4d18-484d-8aba-73b61f6a411e","Type":"ContainerStarted","Data":"85857ac1e7541fa234f5d73910dd5b8ae4d476cd06b64b472f4c585e1d498d6a"} Oct 30 21:48:42.860965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:42.839702 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podStartSLOduration=2.839681278 podStartE2EDuration="2.839681278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:42.83619396 +0000 UTC m=+1874.738058025" watchObservedRunningTime="2026-10-30 21:48:42.839681278 +0000 UTC m=+1874.741545346" Oct 30 21:48:43.162907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:43.155679 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:43.177905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:43.170755 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:43.649823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:43.649787 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:43.650822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:43.650806 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:48:54.235012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.234973 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq"] Oct 30 21:48:54.240285 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.240262 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.324905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.320827 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b188-dockercfg-sw29z\"" Oct 30 21:48:54.335970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.332090 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b188\"" Oct 30 21:48:54.419605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.419572 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq"] Oct 30 21:48:54.466499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.439774 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.466499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.439983 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhxr\" (UniqueName: \"kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr\") pod \"payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.541297 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.541203 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhxr\" (UniqueName: \"kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr\") pod \"payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.541495 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.541461 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.542207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.542184 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.599004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.598971 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhxr\" (UniqueName: \"kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr\") pod \"payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:54.855348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:54.855253 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:55.230896 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:55.230739 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq"] Oct 30 21:48:55.233234 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:48:55.233206 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca56b310_9620_450b_850f_37e8b0673e37.slice/crio-e24a38cf7660205274b84755375807f17596b03e959a856cee40700a5e5753c1 WatchSource:0}: Error finding container e24a38cf7660205274b84755375807f17596b03e959a856cee40700a5e5753c1: Status 404 returned error can't find the container with id e24a38cf7660205274b84755375807f17596b03e959a856cee40700a5e5753c1 Oct 30 21:48:55.820974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:55.820875 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" event={"ID":"ca56b310-9620-450b-850f-37e8b0673e37","Type":"ContainerStarted","Data":"9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c"} Oct 30 21:48:55.821450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:55.820995 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" event={"ID":"ca56b310-9620-450b-850f-37e8b0673e37","Type":"ContainerStarted","Data":"e24a38cf7660205274b84755375807f17596b03e959a856cee40700a5e5753c1"} Oct 30 21:48:55.821450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:55.821024 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:48:55.934775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:48:55.934719 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" podStartSLOduration=1.934703949 podStartE2EDuration="1.934703949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:55.933199997 +0000 UTC m=+1887.835064074" watchObservedRunningTime="2026-10-30 21:48:55.934703949 +0000 UTC m=+1887.836568014" Oct 30 21:49:02.805043 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.804987 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj"] Oct 30 21:49:02.821191 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.821161 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.881323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.878234 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr"] Oct 30 21:49:02.889041 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.887628 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:02.917840 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.917795 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.917869 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918171 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918122 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvfx\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918241 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918315 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918275 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918354 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918456 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918494 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.918596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.918533 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:02.933647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:02.932526 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj"] Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023166 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023215 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023260 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023299 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvfx\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023405 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023441 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023468 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023499 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023585 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023618 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023729 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023762 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023803 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023850 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023916 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023945 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.023978 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5zm8\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.024060 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.025362 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.026913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.025701 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.047916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.047677 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.051100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.050995 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.076978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.063603 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.076978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.064480 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.076978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.064757 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.092978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.092864 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr"] Oct 30 21:49:03.098971 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.098920 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvfx\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.136205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.135523 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.136205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.135820 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.135929 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136095 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136128 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5zm8\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136205 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136195 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136269 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136292 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136331 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136436 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.136948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.136759 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.137504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.137011 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.137504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.137189 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.137504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.137257 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.137749 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.137596 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.141075 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.140587 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.141198 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.141069 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.183070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.183035 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:03.193356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.193327 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.193516 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.193360 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5zm8\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8\") pod \"e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.214118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.214080 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:03.621292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.621251 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj"] Oct 30 21:49:03.622714 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:49:03.622679 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f3aaea_fd91_4970_8bb6_2bc08ecdf22e.slice/crio-c5cfa124107d8e9528b476e7168ee8b3877810ed4cfcc17e443797a5af828e36 WatchSource:0}: Error finding container c5cfa124107d8e9528b476e7168ee8b3877810ed4cfcc17e443797a5af828e36: Status 404 returned error can't find the container with id c5cfa124107d8e9528b476e7168ee8b3877810ed4cfcc17e443797a5af828e36 Oct 30 21:49:03.624987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.624957 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr"] Oct 30 21:49:03.625178 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:49:03.625145 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b41ca1_c2ff_4283_8f28_e7a2199b885f.slice/crio-6295ca7782cdd975cb96f637cd6651951eb28728db372f0764d8abda825a0a00 WatchSource:0}: Error finding container 6295ca7782cdd975cb96f637cd6651951eb28728db372f0764d8abda825a0a00: Status 404 returned error can't find the container with id 6295ca7782cdd975cb96f637cd6651951eb28728db372f0764d8abda825a0a00 Oct 30 21:49:03.625867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.625782 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.625867 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.625851 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.626156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.625937 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.628082 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.628052 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.628172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.628114 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.628172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.628144 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:03.915609 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.915574 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" event={"ID":"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e","Type":"ContainerStarted","Data":"4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81"} Oct 30 21:49:03.916125 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.915634 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" event={"ID":"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e","Type":"ContainerStarted","Data":"c5cfa124107d8e9528b476e7168ee8b3877810ed4cfcc17e443797a5af828e36"} Oct 30 21:49:03.917978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.917944 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" event={"ID":"c4b41ca1-c2ff-4283-8f28-e7a2199b885f","Type":"ContainerStarted","Data":"6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7"} Oct 30 21:49:03.918113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:03.918012 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" event={"ID":"c4b41ca1-c2ff-4283-8f28-e7a2199b885f","Type":"ContainerStarted","Data":"6295ca7782cdd975cb96f637cd6651951eb28728db372f0764d8abda825a0a00"} Oct 30 21:49:04.034164 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.034096 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podStartSLOduration=2.034076289 podStartE2EDuration="2.034076289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:04.032464282 +0000 UTC m=+1895.934328359" watchObservedRunningTime="2026-10-30 21:49:04.034076289 +0000 UTC m=+1895.935940355" Oct 30 21:49:04.150781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.150714 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podStartSLOduration=2.150692018 podStartE2EDuration="2.150692018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:04.147508115 +0000 UTC m=+1896.049372180" watchObservedRunningTime="2026-10-30 21:49:04.150692018 +0000 UTC m=+1896.052556084" Oct 30 21:49:04.183473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.183386 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:04.184857 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.184831 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:04.184955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.184924 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:04.216185 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.216156 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:04.216327 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.216203 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:04.219508 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:04.219484 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:05.195908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:05.187354 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:05.195908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:05.187422 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:05.225872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:05.225075 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:05.225872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:05.225140 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:06.209907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:06.194009 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:06.209907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:06.194116 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:06.226735 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:06.226603 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:06.226735 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:06.226665 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:06.841535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:06.841236 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:49:07.196909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:07.185299 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:07.196909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:07.185366 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:07.221905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:07.216946 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:07.221905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:07.217002 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:08.186907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:08.186331 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:08.186907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:08.186401 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:08.238908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:08.235557 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:08.238908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:08.235620 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:09.213911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.205018 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:09.214145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.205191 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:09.270956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.270779 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:09.270956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.270857 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:09.276908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.273108 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx"] Oct 30 21:49:09.693909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.664186 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.730468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.727836 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d7f4c7\"" Oct 30 21:49:09.733141 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.733100 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vhd\" (UniqueName: \"kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd\") pod \"payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.733485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.733457 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.733962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.733943 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d7f4c7-dockercfg-nmfhw\"" Oct 30 21:49:09.768398 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.768354 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx"] Oct 30 21:49:09.832446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.832410 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz"] Oct 30 21:49:09.835526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.835492 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vhd\" (UniqueName: \"kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd\") pod \"payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.835758 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.835645 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.836601 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.836575 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.840011 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.839094 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:09.841978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.841949 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz"] Oct 30 21:49:09.923662 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.923626 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vhd\" (UniqueName: \"kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd\") pod \"payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:09.937420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.937389 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262g6\" (UniqueName: \"kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6\") pod \"payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:09.937599 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:09.937491 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:10.039503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.039417 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262g6\" (UniqueName: \"kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6\") pod \"payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:10.039688 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.039505 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:10.040256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.040234 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:10.126993 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.126959 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262g6\" (UniqueName: \"kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6\") pod \"payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:10.148822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.148796 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:10.159582 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.159558 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:10.233966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.233220 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:10.233966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.233285 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:10.234507 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.234399 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:49:10.234507 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.234453 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 21:49:10.688910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.688016 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx"] Oct 30 21:49:10.735101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:10.725428 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz"] Oct 30 21:49:10.769804 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:49:10.769751 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf884cd6d_be62_4abe_80a5_555d1de04f61.slice/crio-35fa7872138b103323e3c0d2b3426a2170a395d6da463bcea26ae1ca03fc054b WatchSource:0}: Error finding container 35fa7872138b103323e3c0d2b3426a2170a395d6da463bcea26ae1ca03fc054b: Status 404 returned error can't find the container with id 35fa7872138b103323e3c0d2b3426a2170a395d6da463bcea26ae1ca03fc054b Oct 30 21:49:11.136992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.136662 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" event={"ID":"f884cd6d-be62-4abe-80a5-555d1de04f61","Type":"ContainerStarted","Data":"094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac"} Oct 30 21:49:11.136992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.136742 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" event={"ID":"f884cd6d-be62-4abe-80a5-555d1de04f61","Type":"ContainerStarted","Data":"35fa7872138b103323e3c0d2b3426a2170a395d6da463bcea26ae1ca03fc054b"} Oct 30 21:49:11.136992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.136924 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:11.138791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.138767 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" event={"ID":"d4b87727-6ea9-4613-ba2b-d57b083361f5","Type":"ContainerStarted","Data":"cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b"} Oct 30 21:49:11.138968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.138817 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" event={"ID":"d4b87727-6ea9-4613-ba2b-d57b083361f5","Type":"ContainerStarted","Data":"37231cef3c49697f22a4dfa70654f3f546c0def79530952716e5b51c12e61c92"} Oct 30 21:49:11.138968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.138855 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:11.215508 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.215424 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:49:11.215508 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.215481 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 21:49:11.254424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.254351 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" podStartSLOduration=2.254327814 podStartE2EDuration="2.254327814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:11.252412684 +0000 UTC m=+1903.154276750" watchObservedRunningTime="2026-10-30 21:49:11.254327814 +0000 UTC m=+1903.156191880" Oct 30 21:49:11.282343 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.282310 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:11.283136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.282741 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:11.284120 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.284097 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:49:11.410909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:11.410531 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" podStartSLOduration=2.410512524 podStartE2EDuration="2.410512524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:11.370351537 +0000 UTC m=+1903.272215602" watchObservedRunningTime="2026-10-30 21:49:11.410512524 +0000 UTC m=+1903.312376592" Oct 30 21:49:12.220844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:12.220813 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:12.221281 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:12.221111 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:12.221963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:12.221943 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:49:12.346837 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:49:12.346801 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:49:13.410175 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:13.410138 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6"] Oct 30 21:49:13.410642 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:13.410381 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" containerID="cri-o://0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047" gracePeriod=30 Oct 30 21:49:14.094183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.093685 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq"] Oct 30 21:49:14.117918 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.117731 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.143921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.141224 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.143921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.141319 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbxg\" (UniqueName: \"kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg\") pod \"payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.184650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.180256 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1dc70e\"" Oct 30 21:49:14.184650 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.181092 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1dc70e-dockercfg-d2mqj\"" Oct 30 21:49:14.189767 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.185430 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq"] Oct 30 21:49:14.427440 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.353409 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.428386 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.428347 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbxg\" (UniqueName: \"kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg\") pod \"payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.430090 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.409290 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.478356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.477448 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7"] Oct 30 21:49:14.483587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.483559 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.484241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.483677 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7"] Oct 30 21:49:14.489940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.488497 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbxg\" (UniqueName: \"kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg\") pod \"payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.633292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.633260 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.633445 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.633313 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgrns\" (UniqueName: \"kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns\") pod \"payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.734570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.734480 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.734570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.734543 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgrns\" (UniqueName: \"kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns\") pod \"payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.735309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.735288 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.748945 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.748915 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:14.792477 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.792444 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgrns\" (UniqueName: \"kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns\") pod \"payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:14.832155 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:14.826538 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:15.133978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:15.133947 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq"] Oct 30 21:49:15.138099 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:49:15.138069 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c150d9e_7846_4ebd_82d0_2ed3790a4ecb.slice/crio-9f93856e7f4fcc70d7284741a1ed69c4fda0675bcb4a2819cd53dccbf55f7d83 WatchSource:0}: Error finding container 9f93856e7f4fcc70d7284741a1ed69c4fda0675bcb4a2819cd53dccbf55f7d83: Status 404 returned error can't find the container with id 9f93856e7f4fcc70d7284741a1ed69c4fda0675bcb4a2819cd53dccbf55f7d83 Oct 30 21:49:15.198738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:15.198707 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" event={"ID":"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb","Type":"ContainerStarted","Data":"9f93856e7f4fcc70d7284741a1ed69c4fda0675bcb4a2819cd53dccbf55f7d83"} Oct 30 21:49:15.229092 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:15.229060 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7"] Oct 30 21:49:15.229522 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:49:15.229500 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2ee665_ffed_422b_a220_6304d47f1800.slice/crio-2a643d518845100c621cc96730adad624f51b874e9c78ea83c88301a1aa9c177 WatchSource:0}: Error finding container 2a643d518845100c621cc96730adad624f51b874e9c78ea83c88301a1aa9c177: Status 404 returned error can't find the container with id 2a643d518845100c621cc96730adad624f51b874e9c78ea83c88301a1aa9c177 Oct 30 21:49:16.207174 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.207138 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" event={"ID":"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb","Type":"ContainerStarted","Data":"b6932319ddaf9e61ce370884e5b1d415f9947b75a0c8340fbeecb1a4c32620a4"} Oct 30 21:49:16.207615 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.207237 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:16.208856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.208821 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" event={"ID":"ef2ee665-ffed-422b-a220-6304d47f1800","Type":"ContainerStarted","Data":"2dc83a232ca6e05acc20880a27b80086f4a486bfec14ad8404dcb97352dc85ea"} Oct 30 21:49:16.208997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.208869 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" event={"ID":"ef2ee665-ffed-422b-a220-6304d47f1800","Type":"ContainerStarted","Data":"2a643d518845100c621cc96730adad624f51b874e9c78ea83c88301a1aa9c177"} Oct 30 21:49:16.217920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.217875 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:16.321988 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.321937 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" podStartSLOduration=2.32192184 podStartE2EDuration="2.32192184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.320035634 +0000 UTC m=+1908.221899734" watchObservedRunningTime="2026-10-30 21:49:16.32192184 +0000 UTC m=+1908.223785904" Oct 30 21:49:16.408077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.408034 5914 patch_prober.go:29] interesting pod/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:16.408218 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.408096 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:16.445151 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:16.439681 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" podStartSLOduration=2.439663836 podStartE2EDuration="2.439663836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.438850613 +0000 UTC m=+1908.340714671" watchObservedRunningTime="2026-10-30 21:49:16.439663836 +0000 UTC m=+1908.341527903" Oct 30 21:49:18.702560 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.702531 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:49:18.759248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759210 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759260 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759351 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759382 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759407 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759426 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759497 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759520 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:18.759622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759573 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token\") pod \"79dd13dc-19c6-4439-94f6-128a040d28ab\" (UID: \"79dd13dc-19c6-4439-94f6-128a040d28ab\") " Oct 30 21:49:18.759827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759708 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:18.759827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759788 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:18.760068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759831 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:18.760068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.759986 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:18.760424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.760400 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.760424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.760426 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.760605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.760441 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dd13dc-19c6-4439-94f6-128a040d28ab-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.760605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.760454 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.760605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.760467 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.762001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.761974 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:18.762136 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.762008 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "kube-api-access-lsv4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:18.762463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.762429 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:18.762947 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.762926 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token" pod "79dd13dc-19c6-4439-94f6-128a040d28ab" (UID: "79dd13dc-19c6-4439-94f6-128a040d28ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:18.862472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.862435 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsv4p\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-kube-api-access-lsv4p\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.862472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.862467 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.862472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.862477 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:18.862717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:18.862486 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dd13dc-19c6-4439-94f6-128a040d28ab-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:19.233461 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.233425 5914 generic.go:356] "Generic (PLEG): container finished" podID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerID="0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047" exitCode=0 Oct 30 21:49:19.233669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.233477 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" Oct 30 21:49:19.233669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.233502 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" event={"ID":"79dd13dc-19c6-4439-94f6-128a040d28ab","Type":"ContainerDied","Data":"0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047"} Oct 30 21:49:19.233669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.233557 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6" event={"ID":"79dd13dc-19c6-4439-94f6-128a040d28ab","Type":"ContainerDied","Data":"e4f08ffb5edd8bd18ed6ef8a2e5cee76241c44b3d12581645a10c22c11d205e0"} Oct 30 21:49:19.233669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.233580 5914 scope.go:122] "RemoveContainer" containerID="0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047" Oct 30 21:49:19.246051 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.246028 5914 scope.go:122] "RemoveContainer" containerID="0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047" Oct 30 21:49:19.246366 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:49:19.246346 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047\": container with ID starting with 0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047 not found: ID does not exist" containerID="0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047" Oct 30 21:49:19.246425 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.246376 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047"} err="failed to get container status \"0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047\": rpc error: code = NotFound desc = could not find container \"0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047\": container with ID starting with 0ae74296cf2a95231523eb211ff9551be94c67bd4f83b498bc0ae4049ecd7047 not found: ID does not exist" Oct 30 21:49:19.398963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.398925 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6"] Oct 30 21:49:19.448408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:19.448369 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-f7kc6"] Oct 30 21:49:20.253241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:20.253046 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" path="/var/lib/kubelet/pods/79dd13dc-19c6-4439-94f6-128a040d28ab/volumes" Oct 30 21:49:21.133291 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:21.133256 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv"] Oct 30 21:49:21.133482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:21.133456 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" containerID="cri-o://8ac41409ca9f5f797a907f9cc09d33024e601361c7d33b573f3e72525cffad91" gracePeriod=30 Oct 30 21:49:22.148932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:22.148901 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:49:22.149404 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:22.149219 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:49:25.383565 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:25.383529 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj"] Oct 30 21:49:25.384056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:25.383773 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" containerID="cri-o://6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61" gracePeriod=30 Oct 30 21:49:25.762945 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:25.762848 5914 patch_prober.go:29] interesting pod/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:25.763122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:25.762939 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:26.346121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.346081 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" event={"ID":"45e96d80-8fec-4476-9a30-700da90abc8d","Type":"ContainerDied","Data":"8ac41409ca9f5f797a907f9cc09d33024e601361c7d33b573f3e72525cffad91"} Oct 30 21:49:26.346989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.346949 5914 generic.go:356] "Generic (PLEG): container finished" podID="45e96d80-8fec-4476-9a30-700da90abc8d" containerID="8ac41409ca9f5f797a907f9cc09d33024e601361c7d33b573f3e72525cffad91" exitCode=0 Oct 30 21:49:26.408185 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.408119 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:49:26.488494 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488462 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.488639 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488588 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.488639 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488623 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.488770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488648 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.488770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488673 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.488770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488727 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.488770 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488746 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.489009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488975 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.489009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.488979 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:26.489121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.489087 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:26.489121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.489092 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket\") pod \"45e96d80-8fec-4476-9a30-700da90abc8d\" (UID: \"45e96d80-8fec-4476-9a30-700da90abc8d\") " Oct 30 21:49:26.489227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.489126 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:26.489227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.489171 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:26.489336 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.489301 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:26.490279 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.490250 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.490426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.490283 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.490426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.490299 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45e96d80-8fec-4476-9a30-700da90abc8d-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.490426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.490312 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.490426 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.490327 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.491353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.491330 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:26.491424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.491408 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "kube-api-access-w8v22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:26.491575 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.491551 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:26.491645 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.491560 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy" pod "45e96d80-8fec-4476-9a30-700da90abc8d" (UID: "45e96d80-8fec-4476-9a30-700da90abc8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:26.591712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.591674 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45e96d80-8fec-4476-9a30-700da90abc8d-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.591712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.591709 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45e96d80-8fec-4476-9a30-700da90abc8d-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.591933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.591723 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8v22\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-kube-api-access-w8v22\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:26.591933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:26.591736 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45e96d80-8fec-4476-9a30-700da90abc8d-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:27.220395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.220365 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:49:27.358297 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.358257 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" event={"ID":"45e96d80-8fec-4476-9a30-700da90abc8d","Type":"ContainerDied","Data":"02aad4986e5b937cf4c1d9419e1996056c89a090a86d0c4dc125a8e7afa3751b"} Oct 30 21:49:27.358530 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.358510 5914 scope.go:122] "RemoveContainer" containerID="8ac41409ca9f5f797a907f9cc09d33024e601361c7d33b573f3e72525cffad91" Oct 30 21:49:27.358669 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.358414 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv" Oct 30 21:49:27.485065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.484981 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:49:27.485065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.485052 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:49:27.524937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.524901 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv"] Oct 30 21:49:27.574923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:27.574851 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-p9hwv"] Oct 30 21:49:28.220187 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:28.220152 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" path="/var/lib/kubelet/pods/45e96d80-8fec-4476-9a30-700da90abc8d/volumes" Oct 30 21:49:28.228117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:28.228092 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:49:30.680686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.680657 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:49:30.779134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779099 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779169 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779201 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779262 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779322 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779460 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779488 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779654 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779546 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779661 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy\") pod \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\" (UID: \"b985fa53-55d1-40e6-a488-8a30ef36e1d2\") " Oct 30 21:49:30.779771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779747 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.779912 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.779858 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.780059 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.780037 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:30.780596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.780551 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.781099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.780968 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.781099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.780993 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.781099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.781009 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.781099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.781023 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.781099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.781086 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.782021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.781985 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:49:30.782265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.782245 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "kube-api-access-ffjrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:30.782387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.782360 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:49:30.782480 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.782424 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token" pod "b985fa53-55d1-40e6-a488-8a30ef36e1d2" (UID: "b985fa53-55d1-40e6-a488-8a30ef36e1d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:30.882540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.882507 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjrn\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-kube-api-access-ffjrn\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.882540 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.882542 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.882787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.882556 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.882787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.882565 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:30.882787 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:30.882576 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b985fa53-55d1-40e6-a488-8a30ef36e1d2-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:31.398443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.398408 5914 generic.go:356] "Generic (PLEG): container finished" podID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerID="6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61" exitCode=0 Oct 30 21:49:31.398619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.398468 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" Oct 30 21:49:31.398619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.398490 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" event={"ID":"b985fa53-55d1-40e6-a488-8a30ef36e1d2","Type":"ContainerDied","Data":"6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61"} Oct 30 21:49:31.398619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.398544 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj" event={"ID":"b985fa53-55d1-40e6-a488-8a30ef36e1d2","Type":"ContainerDied","Data":"36602301ab9015afc35f9cd90800051bb44afb7ff4f3f404a3e62c6348feec37"} Oct 30 21:49:31.398619 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.398567 5914 scope.go:122] "RemoveContainer" containerID="6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61" Oct 30 21:49:31.411560 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.411533 5914 scope.go:122] "RemoveContainer" containerID="6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61" Oct 30 21:49:31.411828 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:49:31.411804 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61\": container with ID starting with 6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61 not found: ID does not exist" containerID="6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61" Oct 30 21:49:31.411913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.411842 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61"} err="failed to get container status \"6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61\": rpc error: code = NotFound desc = could not find container \"6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61\": container with ID starting with 6808e38194e53bea64e999d0a5a67123c3bf5b753f532d62e0e882253465ec61 not found: ID does not exist" Oct 30 21:49:31.563198 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.563155 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj"] Oct 30 21:49:31.569630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:31.569601 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-n79sj"] Oct 30 21:49:32.219273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:32.219226 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" path="/var/lib/kubelet/pods/b985fa53-55d1-40e6-a488-8a30ef36e1d2/volumes" Oct 30 21:49:42.347114 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:49:42.347073 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:49:47.064029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:49:47.063993 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:50:02.510150 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510118 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz"] Oct 30 21:50:02.510865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510851 5914 cpu_manager.go:414] "containerMap: removing container" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" Oct 30 21:50:02.510940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510868 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" Oct 30 21:50:02.510940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510903 5914 cpu_manager.go:414] "containerMap: removing container" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" Oct 30 21:50:02.510940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510912 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" Oct 30 21:50:02.511035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510941 5914 cpu_manager.go:414] "containerMap: removing container" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" Oct 30 21:50:02.511035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.510947 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" Oct 30 21:50:02.511099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.511057 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e96d80-8fec-4476-9a30-700da90abc8d" containerName="istio-proxy" Oct 30 21:50:02.511099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.511077 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="79dd13dc-19c6-4439-94f6-128a040d28ab" containerName="istio-proxy" Oct 30 21:50:02.511099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.511084 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b985fa53-55d1-40e6-a488-8a30ef36e1d2" containerName="istio-proxy" Oct 30 21:50:02.515089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.515069 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.568240 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.568178 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-013fb2-serving-cert\"" Oct 30 21:50:02.572800 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.572763 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz"] Oct 30 21:50:02.675997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.675955 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.676240 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.676217 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.676301 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.676259 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwxpc\" (UniqueName: \"kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.676430 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.676410 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.777974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.777841 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.794907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.793005 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwxpc\" (UniqueName: \"kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.794907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.793186 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.794907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.793293 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.795206 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:50:02.794984 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-serving-cert: secret "maas-api-e2e-shared-b-w4-013fb2-serving-cert" not found Oct 30 21:50:02.795206 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:50:02.795051 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls podName:982d2a1c-eacf-41cf-b9d7-649838190b31 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:03.295030502 +0000 UTC m=+1955.196894568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls") pod "maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" (UID: "982d2a1c-eacf-41cf-b9d7-649838190b31") : secret "maas-api-e2e-shared-b-w4-013fb2-serving-cert" not found Oct 30 21:50:02.796377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.796003 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.803908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.800274 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:02.853791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:02.853762 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwxpc\" (UniqueName: \"kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:03.303843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:03.303806 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:03.306511 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:03.306485 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:03.432079 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:03.432045 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:03.814737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:03.814704 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz"] Oct 30 21:50:03.815639 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:50:03.815610 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982d2a1c_eacf_41cf_b9d7_649838190b31.slice/crio-fe3fd48c0105f959ae9298768d5d16db4d0f75dd217d60751c36754c991e2c4e WatchSource:0}: Error finding container fe3fd48c0105f959ae9298768d5d16db4d0f75dd217d60751c36754c991e2c4e: Status 404 returned error can't find the container with id fe3fd48c0105f959ae9298768d5d16db4d0f75dd217d60751c36754c991e2c4e Oct 30 21:50:04.743928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:04.743857 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" event={"ID":"982d2a1c-eacf-41cf-b9d7-649838190b31","Type":"ContainerStarted","Data":"598e132857a810869eb9e8e7889f6c7b3e39efc430220a370a16403564cf9b3a"} Oct 30 21:50:04.744109 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:04.743958 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" event={"ID":"982d2a1c-eacf-41cf-b9d7-649838190b31","Type":"ContainerStarted","Data":"fe3fd48c0105f959ae9298768d5d16db4d0f75dd217d60751c36754c991e2c4e"} Oct 30 21:50:04.744109 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:04.743983 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:04.925820 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:04.925699 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" podStartSLOduration=2.101225153 podStartE2EDuration="2.925684623s" totalImagesPullingTime="824.45947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:04.9235692 +0000 UTC m=+1956.825433264" watchObservedRunningTime="2026-10-30 21:50:04.925684623 +0000 UTC m=+1956.827548727" Oct 30 21:50:10.758138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:10.758107 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:50:12.347578 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:50:12.347539 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:50:32.740110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.740075 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws"] Oct 30 21:50:32.746377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.746353 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.778555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.778528 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.778718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.778645 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmw6\" (UniqueName: \"kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.778761 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.778721 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.778911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.778897 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.801976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.801940 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws"] Oct 30 21:50:32.847149 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.847120 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d7f4c7-serving-cert\"" Oct 30 21:50:32.880735 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.880692 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmw6\" (UniqueName: \"kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.880954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.880802 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.881052 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.881029 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.881123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.881109 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.881184 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:50:32.881153 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-serving-cert: secret "maas-api-e2e-inf-a-d7f4c7-serving-cert" not found Oct 30 21:50:32.881235 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:50:32.881222 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls podName:11f419dc-a8f9-473b-8b76-8fded5bbb494 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:33.381205493 +0000 UTC m=+1985.283069554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls") pod "maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" (UID: "11f419dc-a8f9-473b-8b76-8fded5bbb494") : secret "maas-api-e2e-inf-a-d7f4c7-serving-cert" not found Oct 30 21:50:32.881725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.881696 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.883950 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.883919 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:32.940782 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:32.940740 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmw6\" (UniqueName: \"kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:33.398217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:33.398173 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:33.400846 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:33.400820 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\") pod \"maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:33.662691 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:33.662613 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:34.066001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.063844 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5"] Oct 30 21:50:34.074587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.074556 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5"] Oct 30 21:50:34.074837 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.074655 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.152905 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.141015 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6b42c9\"" Oct 30 21:50:34.203108 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.174478 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.203108 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.174727 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4hx\" (UniqueName: \"kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx\") pod \"payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.315117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.315078 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4hx\" (UniqueName: \"kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx\") pod \"payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.315522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.315499 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.316753 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.316695 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.339543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.319737 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6b42c9-dockercfg-xtrps\"" Oct 30 21:50:34.347675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.346750 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g"] Oct 30 21:50:34.354828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.353681 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g"] Oct 30 21:50:34.354828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.353810 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.383954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.383916 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4hx\" (UniqueName: \"kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx\") pod \"payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.405999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.405964 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:34.521936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.519312 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgn5\" (UniqueName: \"kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5\") pod \"payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.521936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.519450 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.621209 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.621176 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgn5\" (UniqueName: \"kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5\") pod \"payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.622144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.621294 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.622144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.622023 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.647576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.647552 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws"] Oct 30 21:50:34.650897 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:50:34.650854 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f419dc_a8f9_473b_8b76_8fded5bbb494.slice/crio-99788fd700f7bc8979678a20d64aa507aee82e770735d27eaca85ffbb2ac63f0 WatchSource:0}: Error finding container 99788fd700f7bc8979678a20d64aa507aee82e770735d27eaca85ffbb2ac63f0: Status 404 returned error can't find the container with id 99788fd700f7bc8979678a20d64aa507aee82e770735d27eaca85ffbb2ac63f0 Oct 30 21:50:34.679219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.679188 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgn5\" (UniqueName: \"kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5\") pod \"payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.684597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.684575 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:34.792401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:34.792362 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5"] Oct 30 21:50:34.796203 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:50:34.796158 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edca360_fbf2_4f5e_93dd_54e7c7910347.slice/crio-c6e48ae81c46caed0f55c32a0707e6f79969868f870041602ea5f257ebacfb8d WatchSource:0}: Error finding container c6e48ae81c46caed0f55c32a0707e6f79969868f870041602ea5f257ebacfb8d: Status 404 returned error can't find the container with id c6e48ae81c46caed0f55c32a0707e6f79969868f870041602ea5f257ebacfb8d Oct 30 21:50:35.067574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.067544 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g"] Oct 30 21:50:35.068472 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:50:35.068446 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef71a298_f0f5_4284_8059_245296585782.slice/crio-0a7e13a16436df81e7c53c7e077bcd135371c66f6f6223caa8e70b6e6b52496d WatchSource:0}: Error finding container 0a7e13a16436df81e7c53c7e077bcd135371c66f6f6223caa8e70b6e6b52496d: Status 404 returned error can't find the container with id 0a7e13a16436df81e7c53c7e077bcd135371c66f6f6223caa8e70b6e6b52496d Oct 30 21:50:35.142517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.139961 5914 generic.go:356] "Generic (PLEG): container finished" podID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerID="447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe" exitCode=0 Oct 30 21:50:35.142517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.140052 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" event={"ID":"3a7357f2-2289-4074-bf14-fe315ad33e0a","Type":"ContainerDied","Data":"447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe"} Oct 30 21:50:35.142517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.140832 5914 scope.go:122] "RemoveContainer" containerID="447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe" Oct 30 21:50:35.145446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.143511 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:35.145446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.143607 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:35.145446 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.143653 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:50:35.169393 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.169343 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" event={"ID":"ef71a298-f0f5-4284-8059-245296585782","Type":"ContainerStarted","Data":"0a7e13a16436df81e7c53c7e077bcd135371c66f6f6223caa8e70b6e6b52496d"} Oct 30 21:50:35.172936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.171141 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" event={"ID":"11f419dc-a8f9-473b-8b76-8fded5bbb494","Type":"ContainerStarted","Data":"99788fd700f7bc8979678a20d64aa507aee82e770735d27eaca85ffbb2ac63f0"} Oct 30 21:50:35.173084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.172964 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" event={"ID":"0edca360-fbf2-4f5e-93dd-54e7c7910347","Type":"ContainerStarted","Data":"7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe"} Oct 30 21:50:35.173084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.173006 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" event={"ID":"0edca360-fbf2-4f5e-93dd-54e7c7910347","Type":"ContainerStarted","Data":"c6e48ae81c46caed0f55c32a0707e6f79969868f870041602ea5f257ebacfb8d"} Oct 30 21:50:35.173204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.173145 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:35.388297 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:35.388223 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" podStartSLOduration=2.388202631 podStartE2EDuration="2.388202631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:35.374670793 +0000 UTC m=+1987.276534858" watchObservedRunningTime="2026-10-30 21:50:35.388202631 +0000 UTC m=+1987.290066711" Oct 30 21:50:36.359964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.359840 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:36.360451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.360321 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:36.360451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.360362 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" event={"ID":"11f419dc-a8f9-473b-8b76-8fded5bbb494","Type":"ContainerStarted","Data":"a12a22c69cfa133a65dffbcc7547e0fa06de9400750823884ea1620bac6fd877"} Oct 30 21:50:36.360451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.360390 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" event={"ID":"3a7357f2-2289-4074-bf14-fe315ad33e0a","Type":"ContainerStarted","Data":"8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671"} Oct 30 21:50:36.360451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.360417 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" event={"ID":"ef71a298-f0f5-4284-8059-245296585782","Type":"ContainerStarted","Data":"bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c"} Oct 30 21:50:36.429963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.429913 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" podStartSLOduration=3.504808129 podStartE2EDuration="4.42986251s" totalImagesPullingTime="925.054381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:36.42690502 +0000 UTC m=+1988.328769087" watchObservedRunningTime="2026-10-30 21:50:36.42986251 +0000 UTC m=+1988.331726631" Oct 30 21:50:36.604159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.604096 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" podStartSLOduration=2.60407362 podStartE2EDuration="2.60407362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:36.585859555 +0000 UTC m=+1988.487723620" watchObservedRunningTime="2026-10-30 21:50:36.60407362 +0000 UTC m=+1988.505937688" Oct 30 21:50:36.798909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.798011 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:50:36.798909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.798369 5914 patch_prober.go:29] interesting pod/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:50:36.798909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:36.798428 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 21:50:37.597910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:37.594352 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:50:37.798335 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:37.798296 5914 patch_prober.go:29] interesting pod/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:50:37.798525 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:37.798354 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 21:50:38.798007 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:38.797971 5914 patch_prober.go:29] interesting pod/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:50:38.798532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:38.798027 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 21:50:39.798043 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:39.797991 5914 patch_prober.go:29] interesting pod/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:50:39.798483 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:39.798061 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 21:50:40.798415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:40.798374 5914 patch_prober.go:29] interesting pod/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 21:50:40.798840 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:40.798451 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 21:50:41.801323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:41.801290 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:50:41.802480 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:41.802455 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:50:42.345843 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:42.345807 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.115:8443/health\": dial tcp 10.134.0.115:8443: connect: connection refused" Oct 30 21:50:42.347387 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:50:42.347354 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:50:46.340146 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:46.340119 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:50:46.518435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:46.518393 5914 generic.go:356] "Generic (PLEG): container finished" podID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerID="a12a22c69cfa133a65dffbcc7547e0fa06de9400750823884ea1620bac6fd877" exitCode=1 Oct 30 21:50:46.519212 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:46.519184 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" event={"ID":"11f419dc-a8f9-473b-8b76-8fded5bbb494","Type":"ContainerDied","Data":"a12a22c69cfa133a65dffbcc7547e0fa06de9400750823884ea1620bac6fd877"} Oct 30 21:50:46.519953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:46.519931 5914 scope.go:122] "RemoveContainer" containerID="a12a22c69cfa133a65dffbcc7547e0fa06de9400750823884ea1620bac6fd877" Oct 30 21:50:47.347139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:47.347110 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:50:47.529340 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:47.529298 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" event={"ID":"11f419dc-a8f9-473b-8b76-8fded5bbb494","Type":"ContainerStarted","Data":"5f95bb236197d22b5a65ca03bd68d19fca95efae9d80515d58311ef4b8b140c8"} Oct 30 21:50:47.529589 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:47.529565 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:50:53.541508 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:50:53.541477 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:51:04.434599 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.434558 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj"] Oct 30 21:51:04.439621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.439595 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.500768 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.500693 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.500824 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.500855 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.500951 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.501008 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq65t\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.501049 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.501087 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.501110 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501256 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.501171 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.501836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.501805 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9c76ae-openshift-default-dockercfg-rcztt\"" Oct 30 21:51:04.546831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.546799 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj"] Oct 30 21:51:04.632907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.607561 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.663908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.633385 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.664238 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.687097 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.687208 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq65t\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.687268 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.687316 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.687339 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.687434 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.635505 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.688061 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.689408 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.689956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.689778 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.702239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.612587 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.705233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.705198 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.707539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.707504 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.746133 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.746094 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq65t\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.746133 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.746103 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token\") pod \"e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:04.755150 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:04.755126 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:05.191673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.191640 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj"] Oct 30 21:51:05.192639 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:51:05.192608 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b8a9d0_0196_4020_b114_73f3ffa1e7a9.slice/crio-68a7417abe6cd4c5d8aa7132bb18697a54698def1c05b897549d3df5379daba8 WatchSource:0}: Error finding container 68a7417abe6cd4c5d8aa7132bb18697a54698def1c05b897549d3df5379daba8: Status 404 returned error can't find the container with id 68a7417abe6cd4c5d8aa7132bb18697a54698def1c05b897549d3df5379daba8 Oct 30 21:51:05.195745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.195714 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:05.195816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.195783 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:05.195816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.195813 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:05.721074 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.721035 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" event={"ID":"86b8a9d0-0196-4020-b114-73f3ffa1e7a9","Type":"ContainerStarted","Data":"385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208"} Oct 30 21:51:05.721631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.721101 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" event={"ID":"86b8a9d0-0196-4020-b114-73f3ffa1e7a9","Type":"ContainerStarted","Data":"68a7417abe6cd4c5d8aa7132bb18697a54698def1c05b897549d3df5379daba8"} Oct 30 21:51:05.755275 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.755240 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:05.842659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:05.842605 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podStartSLOduration=1.8425888160000001 podStartE2EDuration="1.842588816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:05.841348151 +0000 UTC m=+2017.743212230" watchObservedRunningTime="2026-10-30 21:51:05.842588816 +0000 UTC m=+2017.744452883" Oct 30 21:51:06.761979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:06.761948 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:07.221937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:07.221354 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:51:07.735308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:07.735276 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:07.736361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:07.736335 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:51:12.347563 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:51:12.347525 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:51:12.955438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:12.955404 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj"] Oct 30 21:51:13.006752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.006722 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.038137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.038105 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj"] Oct 30 21:51:13.099909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.071746 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9c76ae\"" Oct 30 21:51:13.099909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.071950 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9c76ae-dockercfg-8h7bc\"" Oct 30 21:51:13.240963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.206086 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.276978 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.263222 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72w9\" (UniqueName: \"kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9\") pod \"payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.411960 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.400402 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72w9\" (UniqueName: \"kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9\") pod \"payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.412801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.412769 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.414127 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.414099 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.419740 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.419714 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw"] Oct 30 21:51:13.427988 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.427962 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.433550 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.433518 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw"] Oct 30 21:51:13.471929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.471873 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72w9\" (UniqueName: \"kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9\") pod \"payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.522541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.519413 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.522541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.519589 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vtf\" (UniqueName: \"kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf\") pod \"payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.579501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.579464 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:13.621751 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.621716 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.622100 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.622072 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vtf\" (UniqueName: \"kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf\") pod \"payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.623114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.623087 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.713917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.711855 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vtf\" (UniqueName: \"kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf\") pod \"payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:13.748813 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:13.748772 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:14.001158 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:51:14.001125 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13b974f_3697_4db8_85df_047cc6c7aab7.slice/crio-e6dc970fcd2d055b70b0c71e80194e437fd47694f6ca3cc9c2058a8b2efd8bd4 WatchSource:0}: Error finding container e6dc970fcd2d055b70b0c71e80194e437fd47694f6ca3cc9c2058a8b2efd8bd4: Status 404 returned error can't find the container with id e6dc970fcd2d055b70b0c71e80194e437fd47694f6ca3cc9c2058a8b2efd8bd4 Oct 30 21:51:14.007837 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:14.003417 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj"] Oct 30 21:51:14.171247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:14.171217 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw"] Oct 30 21:51:14.172480 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:51:14.172452 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6110a5_3d09_4d18_a591_be31f5c5e1d4.slice/crio-48ce95c4fd880e1dc2614d9a5668b03b1da17dc984ba246a41aee981e52da5d3 WatchSource:0}: Error finding container 48ce95c4fd880e1dc2614d9a5668b03b1da17dc984ba246a41aee981e52da5d3: Status 404 returned error can't find the container with id 48ce95c4fd880e1dc2614d9a5668b03b1da17dc984ba246a41aee981e52da5d3 Oct 30 21:51:15.013707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.013667 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" event={"ID":"d13b974f-3697-4db8-85df-047cc6c7aab7","Type":"ContainerStarted","Data":"0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218"} Oct 30 21:51:15.014196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.013935 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:15.014196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.013985 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" event={"ID":"d13b974f-3697-4db8-85df-047cc6c7aab7","Type":"ContainerStarted","Data":"e6dc970fcd2d055b70b0c71e80194e437fd47694f6ca3cc9c2058a8b2efd8bd4"} Oct 30 21:51:15.015667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.015642 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" event={"ID":"ed6110a5-3d09-4d18-a591-be31f5c5e1d4","Type":"ContainerStarted","Data":"d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5"} Oct 30 21:51:15.015780 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.015684 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" event={"ID":"ed6110a5-3d09-4d18-a591-be31f5c5e1d4","Type":"ContainerStarted","Data":"48ce95c4fd880e1dc2614d9a5668b03b1da17dc984ba246a41aee981e52da5d3"} Oct 30 21:51:15.015828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.015783 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:15.132499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.132442 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" podStartSLOduration=3.132424743 podStartE2EDuration="3.132424743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:15.127944181 +0000 UTC m=+2027.029808249" watchObservedRunningTime="2026-10-30 21:51:15.132424743 +0000 UTC m=+2027.034288808" Oct 30 21:51:15.275959 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:15.255194 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" podStartSLOduration=2.25517197 podStartE2EDuration="2.25517197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:15.25183224 +0000 UTC m=+2027.153696306" watchObservedRunningTime="2026-10-30 21:51:15.25517197 +0000 UTC m=+2027.157036037" Oct 30 21:51:20.055942 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:20.055834 5914 generic.go:356] "Generic (PLEG): container finished" podID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerID="a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b" exitCode=0 Oct 30 21:51:20.055942 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:20.055915 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" event={"ID":"aacb3b25-4d18-484d-8aba-73b61f6a411e","Type":"ContainerDied","Data":"a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b"} Oct 30 21:51:20.056428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:20.056413 5914 scope.go:122] "RemoveContainer" containerID="a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b" Oct 30 21:51:20.058282 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:20.058255 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.058354 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:20.058325 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:20.058389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:20.058356 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:51:21.065045 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:21.065005 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" event={"ID":"aacb3b25-4d18-484d-8aba-73b61f6a411e","Type":"ContainerStarted","Data":"193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b"} Oct 30 21:51:21.154458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:21.154422 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:51:21.154664 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:21.154630 5914 patch_prober.go:29] interesting pod/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:51:21.154738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:21.154688 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 21:51:22.154298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:22.154262 5914 patch_prober.go:29] interesting pod/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:51:22.154705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:22.154315 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 21:51:23.155407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:23.155367 5914 patch_prober.go:29] interesting pod/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:51:23.155801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:23.155435 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 21:51:24.154451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:24.154417 5914 patch_prober.go:29] interesting pod/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:51:24.154637 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:24.154468 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 21:51:25.154677 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:25.154644 5914 patch_prober.go:29] interesting pod/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:51:25.155099 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:25.154694 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 21:51:26.027114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:26.027086 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:51:26.027302 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:26.027162 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:51:26.155169 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:26.155127 5914 patch_prober.go:29] interesting pod/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 21:51:26.155552 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:26.155190 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 21:51:27.159400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:27.159363 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:51:27.159924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:27.159771 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:51:27.160564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:27.160545 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:51:29.135940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:29.135900 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l"] Oct 30 21:51:29.136422 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:29.136128 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" containerID="cri-o://1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c" gracePeriod=30 Oct 30 21:51:34.437442 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.437413 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:51:34.545815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.545781 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.545847 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.545867 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546009 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546036 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546089 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546143 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:34.546248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546160 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:34.546248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546218 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546276 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546323 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket\") pod \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\" (UID: \"7a052d83-9021-4ac6-80ea-8b9e0473d3cf\") " Oct 30 21:51:34.546921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546709 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:34.546921 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.546832 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:34.547129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.547104 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:34.547307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.547285 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.547307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.547311 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.547455 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.547328 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.547455 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.547342 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.548670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.548650 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "kube-api-access-n92v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:34.548996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.548974 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:51:34.549156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.549125 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:51:34.549270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.549256 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token" pod "7a052d83-9021-4ac6-80ea-8b9e0473d3cf" (UID: "7a052d83-9021-4ac6-80ea-8b9e0473d3cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:34.649265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.649230 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.649265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.649262 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.649478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.649278 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n92v8\" (UniqueName: \"kubernetes.io/projected/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-kube-api-access-n92v8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.649478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.649293 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.649478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:34.649305 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a052d83-9021-4ac6-80ea-8b9e0473d3cf-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:35.193121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.193083 5914 generic.go:356] "Generic (PLEG): container finished" podID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerID="1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c" exitCode=0 Oct 30 21:51:35.193309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.193143 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" event={"ID":"7a052d83-9021-4ac6-80ea-8b9e0473d3cf","Type":"ContainerDied","Data":"1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c"} Oct 30 21:51:35.193309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.193149 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" Oct 30 21:51:35.193309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.193200 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l" event={"ID":"7a052d83-9021-4ac6-80ea-8b9e0473d3cf","Type":"ContainerDied","Data":"56c08ef5c7d8502f61d1b28196faff42c5d76fff5e8d4165a07906c657aa1307"} Oct 30 21:51:35.193309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.193232 5914 scope.go:122] "RemoveContainer" containerID="1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c" Oct 30 21:51:35.204769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.204750 5914 scope.go:122] "RemoveContainer" containerID="1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c" Oct 30 21:51:35.205052 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:51:35.205031 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c\": container with ID starting with 1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c not found: ID does not exist" containerID="1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c" Oct 30 21:51:35.205129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.205063 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c"} err="failed to get container status \"1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c\": rpc error: code = NotFound desc = could not find container \"1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c\": container with ID starting with 1aa08e22ccbf3c53af876348e854517539f326c0507d044525aef025a841689c not found: ID does not exist" Oct 30 21:51:35.414584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.414550 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l"] Oct 30 21:51:35.455458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:35.455373 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-ll72l"] Oct 30 21:51:36.218445 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:36.218407 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" path="/var/lib/kubelet/pods/7a052d83-9021-4ac6-80ea-8b9e0473d3cf/volumes" Oct 30 21:51:42.350931 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:51:42.350859 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:51:58.941712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:51:58.941673 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq"] Oct 30 21:52:03.942076 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:03.941977 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" podUID="ca56b310-9620-450b-850f-37e8b0673e37" containerName="payload-pre-processing" containerID="cri-o://9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c" gracePeriod=25 Oct 30 21:52:04.300522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.300486 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:52:04.331944 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.331875 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr\" (UniqueName: \"kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr\") pod \"ca56b310-9620-450b-850f-37e8b0673e37\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " Oct 30 21:52:04.332103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.332036 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume\") pod \"ca56b310-9620-450b-850f-37e8b0673e37\" (UID: \"ca56b310-9620-450b-850f-37e8b0673e37\") " Oct 30 21:52:04.332394 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.332363 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume" pod "ca56b310-9620-450b-850f-37e8b0673e37" (UID: "ca56b310-9620-450b-850f-37e8b0673e37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:04.332651 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.332599 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca56b310-9620-450b-850f-37e8b0673e37-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.334159 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.334138 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr" pod "ca56b310-9620-450b-850f-37e8b0673e37" (UID: "ca56b310-9620-450b-850f-37e8b0673e37"). InnerVolumeSpecName "kube-api-access-6xhxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.434412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.434370 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhxr\" (UniqueName: \"kubernetes.io/projected/ca56b310-9620-450b-850f-37e8b0673e37-kube-api-access-6xhxr\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:04.540509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.540406 5914 generic.go:356] "Generic (PLEG): container finished" podID="ca56b310-9620-450b-850f-37e8b0673e37" containerID="9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c" exitCode=0 Oct 30 21:52:04.540509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.540470 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" Oct 30 21:52:04.540509 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.540487 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" event={"ID":"ca56b310-9620-450b-850f-37e8b0673e37","Type":"ContainerDied","Data":"9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c"} Oct 30 21:52:04.540827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.540570 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq" event={"ID":"ca56b310-9620-450b-850f-37e8b0673e37","Type":"ContainerDied","Data":"e24a38cf7660205274b84755375807f17596b03e959a856cee40700a5e5753c1"} Oct 30 21:52:04.540827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.540595 5914 scope.go:122] "RemoveContainer" containerID="9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c" Oct 30 21:52:04.553845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.553820 5914 scope.go:122] "RemoveContainer" containerID="9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c" Oct 30 21:52:04.554143 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:04.554120 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c\": container with ID starting with 9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c not found: ID does not exist" containerID="9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c" Oct 30 21:52:04.554211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.554156 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c"} err="failed to get container status \"9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c\": rpc error: code = NotFound desc = could not find container \"9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c\": container with ID starting with 9eff6370079aa04026aca155740686fbc3d908762287e92121c79ec2c162bf2c not found: ID does not exist" Oct 30 21:52:04.710556 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.710512 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq"] Oct 30 21:52:04.716269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:04.716231 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b188-57b95d87b7-9t9rq"] Oct 30 21:52:06.219283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:06.219247 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca56b310-9620-450b-850f-37e8b0673e37" path="/var/lib/kubelet/pods/ca56b310-9620-450b-850f-37e8b0673e37/volumes" Oct 30 21:52:12.346521 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:12.346483 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:52:21.055763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:21.055718 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz"] Oct 30 21:52:21.168042 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:21.168012 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx"] Oct 30 21:52:26.056348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.056303 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" podUID="f884cd6d-be62-4abe-80a5-555d1de04f61" containerName="payload-processing" containerID="cri-o://094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac" gracePeriod=25 Oct 30 21:52:26.168961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.168913 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" podUID="d4b87727-6ea9-4613-ba2b-d57b083361f5" containerName="payload-pre-processing" containerID="cri-o://cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b" gracePeriod=25 Oct 30 21:52:26.367699 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.365279 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:52:26.434615 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.434580 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6\" (UniqueName: \"kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6\") pod \"f884cd6d-be62-4abe-80a5-555d1de04f61\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " Oct 30 21:52:26.434837 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.434816 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume\") pod \"f884cd6d-be62-4abe-80a5-555d1de04f61\" (UID: \"f884cd6d-be62-4abe-80a5-555d1de04f61\") " Oct 30 21:52:26.435294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.435262 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume" pod "f884cd6d-be62-4abe-80a5-555d1de04f61" (UID: "f884cd6d-be62-4abe-80a5-555d1de04f61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:26.435597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.435570 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f884cd6d-be62-4abe-80a5-555d1de04f61-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:26.437693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.437669 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6" pod "f884cd6d-be62-4abe-80a5-555d1de04f61" (UID: "f884cd6d-be62-4abe-80a5-555d1de04f61"). InnerVolumeSpecName "kube-api-access-262g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:26.464003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.463971 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:52:26.536797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.536759 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume\") pod \"d4b87727-6ea9-4613-ba2b-d57b083361f5\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " Oct 30 21:52:26.536987 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.536967 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd\" (UniqueName: \"kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd\") pod \"d4b87727-6ea9-4613-ba2b-d57b083361f5\" (UID: \"d4b87727-6ea9-4613-ba2b-d57b083361f5\") " Oct 30 21:52:26.537222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.537190 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume" pod "d4b87727-6ea9-4613-ba2b-d57b083361f5" (UID: "d4b87727-6ea9-4613-ba2b-d57b083361f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:26.537798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.537695 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262g6\" (UniqueName: \"kubernetes.io/projected/f884cd6d-be62-4abe-80a5-555d1de04f61-kube-api-access-262g6\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:26.537798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.537716 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b87727-6ea9-4613-ba2b-d57b083361f5-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:26.539401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.539372 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd" pod "d4b87727-6ea9-4613-ba2b-d57b083361f5" (UID: "d4b87727-6ea9-4613-ba2b-d57b083361f5"). InnerVolumeSpecName "kube-api-access-m8vhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:26.639548 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.639514 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8vhd\" (UniqueName: \"kubernetes.io/projected/d4b87727-6ea9-4613-ba2b-d57b083361f5-kube-api-access-m8vhd\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:26.760225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.760174 5914 generic.go:356] "Generic (PLEG): container finished" podID="d4b87727-6ea9-4613-ba2b-d57b083361f5" containerID="cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b" exitCode=0 Oct 30 21:52:26.760468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.760244 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" Oct 30 21:52:26.760468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.760284 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" event={"ID":"d4b87727-6ea9-4613-ba2b-d57b083361f5","Type":"ContainerDied","Data":"cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b"} Oct 30 21:52:26.760468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.760385 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx" event={"ID":"d4b87727-6ea9-4613-ba2b-d57b083361f5","Type":"ContainerDied","Data":"37231cef3c49697f22a4dfa70654f3f546c0def79530952716e5b51c12e61c92"} Oct 30 21:52:26.760468 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.760421 5914 scope.go:122] "RemoveContainer" containerID="cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b" Oct 30 21:52:26.762414 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.762387 5914 generic.go:356] "Generic (PLEG): container finished" podID="f884cd6d-be62-4abe-80a5-555d1de04f61" containerID="094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac" exitCode=0 Oct 30 21:52:26.762539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.762431 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" event={"ID":"f884cd6d-be62-4abe-80a5-555d1de04f61","Type":"ContainerDied","Data":"094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac"} Oct 30 21:52:26.762539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.762439 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" Oct 30 21:52:26.762539 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.762467 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz" event={"ID":"f884cd6d-be62-4abe-80a5-555d1de04f61","Type":"ContainerDied","Data":"35fa7872138b103323e3c0d2b3426a2170a395d6da463bcea26ae1ca03fc054b"} Oct 30 21:52:26.772213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.772192 5914 scope.go:122] "RemoveContainer" containerID="cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b" Oct 30 21:52:26.772540 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:26.772519 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b\": container with ID starting with cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b not found: ID does not exist" containerID="cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b" Oct 30 21:52:26.772605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.772548 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b"} err="failed to get container status \"cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b\": rpc error: code = NotFound desc = could not find container \"cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b\": container with ID starting with cdd77a06238b7bd2227a5995ca1fc36c139e04b31b4d98fce3188d9d0482fe8b not found: ID does not exist" Oct 30 21:52:26.772605 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.772567 5914 scope.go:122] "RemoveContainer" containerID="094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac" Oct 30 21:52:26.783586 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.783568 5914 scope.go:122] "RemoveContainer" containerID="094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac" Oct 30 21:52:26.783935 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:26.783904 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac\": container with ID starting with 094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac not found: ID does not exist" containerID="094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac" Oct 30 21:52:26.784000 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.783945 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac"} err="failed to get container status \"094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac\": rpc error: code = NotFound desc = could not find container \"094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac\": container with ID starting with 094b5f5ca948c3ef8e7eaa8a91d9b0e51ea489aa0cfe4bcc55b7389f3fd71bac not found: ID does not exist" Oct 30 21:52:26.927299 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.927212 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz"] Oct 30 21:52:26.930781 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:26.930756 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d7f4c7-86584cbcdc-w5jqz"] Oct 30 21:52:27.041299 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:27.041263 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx"] Oct 30 21:52:27.044666 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:27.044639 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d7f4c7-794784bc55-nf6bx"] Oct 30 21:52:28.219435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:28.219397 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b87727-6ea9-4613-ba2b-d57b083361f5" path="/var/lib/kubelet/pods/d4b87727-6ea9-4613-ba2b-d57b083361f5/volumes" Oct 30 21:52:28.220030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:28.220010 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f884cd6d-be62-4abe-80a5-555d1de04f61" path="/var/lib/kubelet/pods/f884cd6d-be62-4abe-80a5-555d1de04f61/volumes" Oct 30 21:52:29.189581 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:29.189546 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:52:29.190780 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:29.190756 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:52:36.678964 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:36.678927 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:52:42.347400 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:42.347362 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:52:56.231443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.231412 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j"] Oct 30 21:52:56.232158 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232143 5914 cpu_manager.go:414] "containerMap: removing container" podUID="d4b87727-6ea9-4613-ba2b-d57b083361f5" containerName="payload-pre-processing" Oct 30 21:52:56.232214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232159 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b87727-6ea9-4613-ba2b-d57b083361f5" containerName="payload-pre-processing" Oct 30 21:52:56.232214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232178 5914 cpu_manager.go:414] "containerMap: removing container" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" Oct 30 21:52:56.232214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232183 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" Oct 30 21:52:56.232214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232206 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ca56b310-9620-450b-850f-37e8b0673e37" containerName="payload-pre-processing" Oct 30 21:52:56.232214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232211 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca56b310-9620-450b-850f-37e8b0673e37" containerName="payload-pre-processing" Oct 30 21:52:56.232370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232218 5914 cpu_manager.go:414] "containerMap: removing container" podUID="f884cd6d-be62-4abe-80a5-555d1de04f61" containerName="payload-processing" Oct 30 21:52:56.232370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232222 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f884cd6d-be62-4abe-80a5-555d1de04f61" containerName="payload-processing" Oct 30 21:52:56.232370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232330 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca56b310-9620-450b-850f-37e8b0673e37" containerName="payload-pre-processing" Oct 30 21:52:56.232370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232340 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="f884cd6d-be62-4abe-80a5-555d1de04f61" containerName="payload-processing" Oct 30 21:52:56.232370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232352 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b87727-6ea9-4613-ba2b-d57b083361f5" containerName="payload-pre-processing" Oct 30 21:52:56.232370 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.232359 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a052d83-9021-4ac6-80ea-8b9e0473d3cf" containerName="istio-proxy" Oct 30 21:52:56.236036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.236019 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.339340 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.339302 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9c76ae-serving-cert\"" Oct 30 21:52:56.342122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.342090 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.342263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.342154 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84cw\" (UniqueName: \"kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.342517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.342490 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.342643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.342626 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.343180 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.343163 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j"] Oct 30 21:52:56.444233 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.444196 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.444418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.444291 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.444418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.444339 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.444418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.444384 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84cw\" (UniqueName: \"kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.444596 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:56.444531 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-serving-cert: secret "maas-api-e2e-inf-a-9c76ae-serving-cert" not found Oct 30 21:52:56.444653 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:52:56.444632 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls podName:59bbd2ec-0a1e-406a-945a-a271d0e66bc5 nodeName:}" failed. No retries permitted until 2026-10-30 21:52:56.944608863 +0000 UTC m=+2128.846472919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls") pod "maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" (UID: "59bbd2ec-0a1e-406a-945a-a271d0e66bc5") : secret "maas-api-e2e-inf-a-9c76ae-serving-cert" not found Oct 30 21:52:56.445067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.445042 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.446871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.446855 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.523969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.523816 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84cw\" (UniqueName: \"kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.967915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.967849 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:56.970519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:56.970494 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\") pod \"maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:57.151061 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:57.151029 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:57.545375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:57.545346 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j"] Oct 30 21:52:57.545852 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:52:57.545702 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bbd2ec_0a1e_406a_945a_a271d0e66bc5.slice/crio-0e7b1c74667ac999c67460e52c0cae2dc91c96bd952f4fbb0ab4f1977cc9e277 WatchSource:0}: Error finding container 0e7b1c74667ac999c67460e52c0cae2dc91c96bd952f4fbb0ab4f1977cc9e277: Status 404 returned error can't find the container with id 0e7b1c74667ac999c67460e52c0cae2dc91c96bd952f4fbb0ab4f1977cc9e277 Oct 30 21:52:57.547412 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:57.547389 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:52:57.990792 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:57.990753 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" event={"ID":"59bbd2ec-0a1e-406a-945a-a271d0e66bc5","Type":"ContainerStarted","Data":"0e7b1c74667ac999c67460e52c0cae2dc91c96bd952f4fbb0ab4f1977cc9e277"} Oct 30 21:52:59.000696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:59.000661 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" event={"ID":"59bbd2ec-0a1e-406a-945a-a271d0e66bc5","Type":"ContainerStarted","Data":"70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125"} Oct 30 21:52:59.001168 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:59.000762 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:52:59.163219 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:52:59.163163 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" podStartSLOduration=2.366610566 podStartE2EDuration="3.163148364s" totalImagesPullingTime="796.537798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:59.159288327 +0000 UTC m=+2131.061152394" watchObservedRunningTime="2026-10-30 21:52:59.163148364 +0000 UTC m=+2131.065012420" Oct 30 21:53:05.015720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:05.015685 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:53:10.447355 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:10.447308 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4"] Oct 30 21:53:10.448033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:10.447589 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" podUID="1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" containerName="maas-api" containerID="cri-o://688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8" gracePeriod=30 Oct 30 21:53:10.837942 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:10.824773 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:53:11.026115 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.026068 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5\" (UniqueName: \"kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5\") pod \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " Oct 30 21:53:11.026329 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.026229 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\") pod \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " Oct 30 21:53:11.026329 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.026323 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca\") pod \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " Oct 30 21:53:11.026493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.026473 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs\") pod \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\" (UID: \"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9\") " Oct 30 21:53:11.026996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.026731 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca" pod "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" (UID: "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:11.027829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.027803 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.029287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.029076 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs" pod "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" (UID: "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:11.029707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.029632 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5" pod "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" (UID: "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9"). InnerVolumeSpecName "kube-api-access-zc8d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:11.032591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.032220 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls" pod "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" (UID: "1d81c1ad-ba94-4417-a066-ae7a1e74c2b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:11.091718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.091682 5914 generic.go:356] "Generic (PLEG): container finished" podID="1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" containerID="688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8" exitCode=0 Oct 30 21:53:11.091939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.091731 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" Oct 30 21:53:11.091939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.091768 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" event={"ID":"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9","Type":"ContainerDied","Data":"688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8"} Oct 30 21:53:11.091939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.091828 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4" event={"ID":"1d81c1ad-ba94-4417-a066-ae7a1e74c2b9","Type":"ContainerDied","Data":"22c0ee4c33aea4d770cd3c276cf6508bfc87ea01d711e098e10c956066e657df"} Oct 30 21:53:11.091939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.091860 5914 scope.go:122] "RemoveContainer" containerID="688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8" Oct 30 21:53:11.103786 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.103765 5914 scope.go:122] "RemoveContainer" containerID="688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8" Oct 30 21:53:11.104146 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:11.104120 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8\": container with ID starting with 688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8 not found: ID does not exist" containerID="688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8" Oct 30 21:53:11.104221 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.104155 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8"} err="failed to get container status \"688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8\": rpc error: code = NotFound desc = could not find container \"688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8\": container with ID starting with 688adeeb9b8e8a0d1f80eb4376b40b5fe3d581efdce528c243ab379bbebd28b8 not found: ID does not exist" Oct 30 21:53:11.129387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.129350 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.129387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.129386 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8d5\" (UniqueName: \"kubernetes.io/projected/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-kube-api-access-zc8d5\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.129621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.129397 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:11.261356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.261307 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4"] Oct 30 21:53:11.269231 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:11.269185 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-218800-5c4b85f749-4qmf4"] Oct 30 21:53:12.219102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:12.219069 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" path="/var/lib/kubelet/pods/1d81c1ad-ba94-4417-a066-ae7a1e74c2b9/volumes" Oct 30 21:53:12.348510 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:12.348470 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:53:13.385008 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:13.384969 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4"] Oct 30 21:53:13.385531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:13.385172 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" containerID="cri-o://3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706" gracePeriod=30 Oct 30 21:53:18.123320 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.123279 5914 patch_prober.go:29] interesting pod/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:53:18.123738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.123350 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:53:18.678259 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.678224 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:53:18.765658 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.765618 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.765835 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.765812 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.765946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.765868 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.765975 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766085 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.765874 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:18.766170 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766117 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766183 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766244 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766308 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766419 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766337 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy\") pod \"58507410-d70e-4a94-8e65-9861b2d7bf99\" (UID: \"58507410-d70e-4a94-8e65-9861b2d7bf99\") " Oct 30 21:53:18.766536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766454 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:18.766536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766474 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:18.766536 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.766490 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:18.767225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.767092 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.767225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.767119 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.767225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.767134 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.767225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.767149 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.767225 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.767090 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:18.769121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.769095 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "kube-api-access-cmdml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:18.769278 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.769204 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:18.769278 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.769267 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:53:18.769394 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.769290 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token" pod "58507410-d70e-4a94-8e65-9861b2d7bf99" (UID: "58507410-d70e-4a94-8e65-9861b2d7bf99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:18.868762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.868729 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.868762 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.868762 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58507410-d70e-4a94-8e65-9861b2d7bf99-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.869015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.868776 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmdml\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-kube-api-access-cmdml\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.869015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.868790 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:18.869015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:18.868804 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58507410-d70e-4a94-8e65-9861b2d7bf99-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:19.157435 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.157395 5914 generic.go:356] "Generic (PLEG): container finished" podID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerID="3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706" exitCode=0 Oct 30 21:53:19.157995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.157449 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" Oct 30 21:53:19.157995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.157482 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" event={"ID":"58507410-d70e-4a94-8e65-9861b2d7bf99","Type":"ContainerDied","Data":"3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706"} Oct 30 21:53:19.157995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.157558 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4" event={"ID":"58507410-d70e-4a94-8e65-9861b2d7bf99","Type":"ContainerDied","Data":"935b950c507ed7f2680ea1f256aa8574cb9be27c1bfee8b831f394f1946aa86b"} Oct 30 21:53:19.157995 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.157590 5914 scope.go:122] "RemoveContainer" containerID="3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706" Oct 30 21:53:19.169597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.169535 5914 scope.go:122] "RemoveContainer" containerID="3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706" Oct 30 21:53:19.169836 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:19.169816 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706\": container with ID starting with 3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706 not found: ID does not exist" containerID="3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706" Oct 30 21:53:19.169928 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.169845 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706"} err="failed to get container status \"3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706\": rpc error: code = NotFound desc = could not find container \"3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706\": container with ID starting with 3b175fc50b9d0d559179f49c75357e7046285ddc70cf0a435a14c7ac8ed42706 not found: ID does not exist" Oct 30 21:53:19.338476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.338430 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4"] Oct 30 21:53:19.341816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:19.341784 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-218800-openshift-default-7888c6d55c-dpqs4"] Oct 30 21:53:20.227532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:20.227500 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" path="/var/lib/kubelet/pods/58507410-d70e-4a94-8e65-9861b2d7bf99/volumes" Oct 30 21:53:32.591476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:32.591444 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn"] Oct 30 21:53:32.591946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:32.591649 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" containerID="cri-o://8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671" gracePeriod=30 Oct 30 21:53:33.134466 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:33.134430 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:53:37.934791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:37.934763 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:53:38.110448 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.110407 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.110713 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.110693 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.110920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.110903 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.111141 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.111123 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.111261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.111246 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.111395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.111380 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.111517 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.111505 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.111678 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.111665 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.111816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.111804 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy\") pod \"3a7357f2-2289-4074-bf14-fe315ad33e0a\" (UID: \"3a7357f2-2289-4074-bf14-fe315ad33e0a\") " Oct 30 21:53:38.117133 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.117097 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:38.117839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.117815 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:38.118339 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.118310 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:38.119152 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.119123 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:38.119363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.119333 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:38.119511 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.119464 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:38.119821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.119652 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:38.120362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.120306 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "kube-api-access-tbfj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:38.121362 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.121341 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo" pod "3a7357f2-2289-4074-bf14-fe315ad33e0a" (UID: "3a7357f2-2289-4074-bf14-fe315ad33e0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:53:38.214416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.214325 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.214632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.214616 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.214741 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.214728 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a7357f2-2289-4074-bf14-fe315ad33e0a-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.214830 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.214818 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfj8\" (UniqueName: \"kubernetes.io/projected/3a7357f2-2289-4074-bf14-fe315ad33e0a-kube-api-access-tbfj8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.214932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.214920 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.215026 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.215015 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.215110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.215099 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.215193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.215182 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.215273 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.215262 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a7357f2-2289-4074-bf14-fe315ad33e0a-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:38.366759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.366726 5914 generic.go:356] "Generic (PLEG): container finished" podID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerID="8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671" exitCode=0 Oct 30 21:53:38.366939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.366786 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" Oct 30 21:53:38.366939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.366813 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" event={"ID":"3a7357f2-2289-4074-bf14-fe315ad33e0a","Type":"ContainerDied","Data":"8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671"} Oct 30 21:53:38.366939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.366894 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn" event={"ID":"3a7357f2-2289-4074-bf14-fe315ad33e0a","Type":"ContainerDied","Data":"23855262569eeab91d934768394a73b817133896ed1da5c03c425eba0e8e56b8"} Oct 30 21:53:38.366939 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.366925 5914 scope.go:122] "RemoveContainer" containerID="8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671" Oct 30 21:53:38.404943 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.404913 5914 scope.go:122] "RemoveContainer" containerID="447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe" Oct 30 21:53:38.419185 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.419162 5914 scope.go:122] "RemoveContainer" containerID="8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671" Oct 30 21:53:38.419561 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:38.419541 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671\": container with ID starting with 8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671 not found: ID does not exist" containerID="8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671" Oct 30 21:53:38.419611 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.419571 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671"} err="failed to get container status \"8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671\": rpc error: code = NotFound desc = could not find container \"8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671\": container with ID starting with 8b7f4ba9153ecf5e55586129bd313dfce136a6686ba28f80468c604f0ced6671 not found: ID does not exist" Oct 30 21:53:38.419611 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.419590 5914 scope.go:122] "RemoveContainer" containerID="447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe" Oct 30 21:53:38.419976 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:38.419869 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe\": container with ID starting with 447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe not found: ID does not exist" containerID="447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe" Oct 30 21:53:38.420021 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.419989 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe"} err="failed to get container status \"447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe\": rpc error: code = NotFound desc = could not find container \"447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe\": container with ID starting with 447f8e17985f978844f2c0c565245fe11c5ffd07fbc13b332bb089a126101ffe not found: ID does not exist" Oct 30 21:53:38.542768 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.542689 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn"] Oct 30 21:53:38.600602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:38.600560 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4a1551-openshift-default-5866f4745f-rn9dn"] Oct 30 21:53:40.219538 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:40.219507 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" path="/var/lib/kubelet/pods/3a7357f2-2289-4074-bf14-fe315ad33e0a/volumes" Oct 30 21:53:42.355643 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:42.355585 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:53:45.452039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:45.452008 5914 generic.go:356] "Generic (PLEG): container finished" podID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerID="385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208" exitCode=0 Oct 30 21:53:45.452492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:45.452087 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" event={"ID":"86b8a9d0-0196-4020-b114-73f3ffa1e7a9","Type":"ContainerDied","Data":"385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208"} Oct 30 21:53:45.452564 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:45.452549 5914 scope.go:122] "RemoveContainer" containerID="385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208" Oct 30 21:53:45.454384 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:45.454354 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:45.454481 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:45.454424 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:45.454481 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:45.454455 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:46.461411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:46.461330 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" event={"ID":"86b8a9d0-0196-4020-b114-73f3ffa1e7a9","Type":"ContainerStarted","Data":"e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13"} Oct 30 21:53:46.755991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:46.755908 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:53:46.756165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:46.756141 5914 patch_prober.go:29] interesting pod/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:53:46.756239 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:46.756219 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 21:53:47.756211 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:47.756173 5914 patch_prober.go:29] interesting pod/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:53:47.756613 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:47.756232 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 21:53:48.755703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:48.755663 5914 patch_prober.go:29] interesting pod/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:53:48.755909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:48.755722 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 21:53:49.756013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:49.755973 5914 patch_prober.go:29] interesting pod/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:53:49.756405 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:49.756034 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 21:53:50.102031 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:50.101952 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6"] Oct 30 21:53:50.102407 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:50.102377 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" containerID="cri-o://193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b" gracePeriod=30 Oct 30 21:53:50.755499 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:50.755467 5914 patch_prober.go:29] interesting pod/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:53:50.755700 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:50.755533 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 21:53:51.756476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:51.756438 5914 patch_prober.go:29] interesting pod/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 21:53:51.756875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:51.756506 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 21:53:52.735485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:52.735454 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:53:52.761145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:52.761115 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:53:53.544766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:53.544742 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:53:55.384985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.384961 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:53:55.519110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519077 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519146 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519179 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519220 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519351 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:55.519454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519435 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:55.519572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519459 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519494 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519524 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519572 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519567 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519622 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr\") pod \"aacb3b25-4d18-484d-8aba-73b61f6a411e\" (UID: \"aacb3b25-4d18-484d-8aba-73b61f6a411e\") " Oct 30 21:53:55.519784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519559 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:55.519784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519705 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:55.519975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.519927 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:55.521147 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.521115 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.521284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.521172 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.521284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.521190 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacb3b25-4d18-484d-8aba-73b61f6a411e-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.521284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.521206 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.521284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.521237 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.531087 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.531052 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:53:55.531267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.531143 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:55.531353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.531141 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "kube-api-access-74mmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:55.533248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.533207 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token" pod "aacb3b25-4d18-484d-8aba-73b61f6a411e" (UID: "aacb3b25-4d18-484d-8aba-73b61f6a411e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:55.561648 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.561613 5914 generic.go:356] "Generic (PLEG): container finished" podID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerID="193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b" exitCode=0 Oct 30 21:53:55.561816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.561686 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" Oct 30 21:53:55.561816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.561692 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" event={"ID":"aacb3b25-4d18-484d-8aba-73b61f6a411e","Type":"ContainerDied","Data":"193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b"} Oct 30 21:53:55.561816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.561772 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6" event={"ID":"aacb3b25-4d18-484d-8aba-73b61f6a411e","Type":"ContainerDied","Data":"85857ac1e7541fa234f5d73910dd5b8ae4d476cd06b64b472f4c585e1d498d6a"} Oct 30 21:53:55.561816 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.561795 5914 scope.go:122] "RemoveContainer" containerID="193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b" Oct 30 21:53:55.575056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.575032 5914 scope.go:122] "RemoveContainer" containerID="a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b" Oct 30 21:53:55.585437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.585414 5914 scope.go:122] "RemoveContainer" containerID="193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b" Oct 30 21:53:55.585794 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:55.585773 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b\": container with ID starting with 193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b not found: ID does not exist" containerID="193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b" Oct 30 21:53:55.585913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.585801 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b"} err="failed to get container status \"193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b\": rpc error: code = NotFound desc = could not find container \"193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b\": container with ID starting with 193ab3478488a32fb9b525dd0c12992f1176a7ec3d9ebd9e2b8ad323961a115b not found: ID does not exist" Oct 30 21:53:55.585913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.585819 5914 scope.go:122] "RemoveContainer" containerID="a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b" Oct 30 21:53:55.586161 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:53:55.586134 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b\": container with ID starting with a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b not found: ID does not exist" containerID="a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b" Oct 30 21:53:55.586227 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.586171 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b"} err="failed to get container status \"a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b\": rpc error: code = NotFound desc = could not find container \"a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b\": container with ID starting with a41334dcf1307a23c5476d59fbc202b99d1e633a962cc88342b4a8c51d60103b not found: ID does not exist" Oct 30 21:53:55.623067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.623038 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.623067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.623062 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.623254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.623072 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mmr\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-kube-api-access-74mmr\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.623254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.623083 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacb3b25-4d18-484d-8aba-73b61f6a411e-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:55.728738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.728661 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6"] Oct 30 21:53:55.733912 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:55.733854 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50b188-openshift-default-768d545959-5ksg6"] Oct 30 21:53:56.218724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:56.218690 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" path="/var/lib/kubelet/pods/aacb3b25-4d18-484d-8aba-73b61f6a411e/volumes" Oct 30 21:53:57.465856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:57.465816 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws"] Oct 30 21:53:57.466710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:57.466652 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" containerID="cri-o://5f95bb236197d22b5a65ca03bd68d19fca95efae9d80515d58311ef4b8b140c8" gracePeriod=30 Oct 30 21:53:57.665087 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:57.664927 5914 generic.go:356] "Generic (PLEG): container finished" podID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerID="5f95bb236197d22b5a65ca03bd68d19fca95efae9d80515d58311ef4b8b140c8" exitCode=0 Oct 30 21:53:57.665087 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:57.665068 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" event={"ID":"11f419dc-a8f9-473b-8b76-8fded5bbb494","Type":"ContainerDied","Data":"5f95bb236197d22b5a65ca03bd68d19fca95efae9d80515d58311ef4b8b140c8"} Oct 30 21:53:57.665368 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:57.665137 5914 scope.go:122] "RemoveContainer" containerID="a12a22c69cfa133a65dffbcc7547e0fa06de9400750823884ea1620bac6fd877" Oct 30 21:53:57.995908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:57.989858 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:53:58.096476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.096391 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\") pod \"11f419dc-a8f9-473b-8b76-8fded5bbb494\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " Oct 30 21:53:58.096476 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.096451 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6\" (UniqueName: \"kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6\") pod \"11f419dc-a8f9-473b-8b76-8fded5bbb494\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " Oct 30 21:53:58.096736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.096539 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs\") pod \"11f419dc-a8f9-473b-8b76-8fded5bbb494\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " Oct 30 21:53:58.096736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.096626 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca\") pod \"11f419dc-a8f9-473b-8b76-8fded5bbb494\" (UID: \"11f419dc-a8f9-473b-8b76-8fded5bbb494\") " Oct 30 21:53:58.097053 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.097025 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca" pod "11f419dc-a8f9-473b-8b76-8fded5bbb494" (UID: "11f419dc-a8f9-473b-8b76-8fded5bbb494"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:58.097296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.097263 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f419dc-a8f9-473b-8b76-8fded5bbb494-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:58.098864 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.098839 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6" pod "11f419dc-a8f9-473b-8b76-8fded5bbb494" (UID: "11f419dc-a8f9-473b-8b76-8fded5bbb494"). InnerVolumeSpecName "kube-api-access-wrmw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:58.099373 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.099344 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls" pod "11f419dc-a8f9-473b-8b76-8fded5bbb494" (UID: "11f419dc-a8f9-473b-8b76-8fded5bbb494"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:58.099475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.099429 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs" pod "11f419dc-a8f9-473b-8b76-8fded5bbb494" (UID: "11f419dc-a8f9-473b-8b76-8fded5bbb494"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:58.198865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.198834 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:58.198865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.198863 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrmw6\" (UniqueName: \"kubernetes.io/projected/11f419dc-a8f9-473b-8b76-8fded5bbb494-kube-api-access-wrmw6\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:58.199073 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.198873 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11f419dc-a8f9-473b-8b76-8fded5bbb494-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:53:58.673500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.673466 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" Oct 30 21:53:58.673500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.673486 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws" event={"ID":"11f419dc-a8f9-473b-8b76-8fded5bbb494","Type":"ContainerDied","Data":"99788fd700f7bc8979678a20d64aa507aee82e770735d27eaca85ffbb2ac63f0"} Oct 30 21:53:58.674047 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.673565 5914 scope.go:122] "RemoveContainer" containerID="5f95bb236197d22b5a65ca03bd68d19fca95efae9d80515d58311ef4b8b140c8" Oct 30 21:53:58.839460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.839423 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws"] Oct 30 21:53:58.889526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:53:58.889492 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d7f4c7-7cbf5bdffc-fdrws"] Oct 30 21:54:00.219492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:00.219458 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" path="/var/lib/kubelet/pods/11f419dc-a8f9-473b-8b76-8fded5bbb494/volumes" Oct 30 21:54:03.802348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.802314 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2"] Oct 30 21:54:03.803077 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803060 5914 cpu_manager.go:414] "containerMap: removing container" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803077 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803100 5914 cpu_manager.go:414] "containerMap: removing container" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803107 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803118 5914 cpu_manager.go:414] "containerMap: removing container" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803125 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803144 5914 cpu_manager.go:414] "containerMap: removing container" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803148 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803160 5914 cpu_manager.go:414] "containerMap: removing container" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803165 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803183 5914 cpu_manager.go:414] "containerMap: removing container" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" Oct 30 21:54:03.803194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803187 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803201 5914 cpu_manager.go:414] "containerMap: removing container" podUID="1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" containerName="maas-api" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803205 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" containerName="maas-api" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803216 5914 cpu_manager.go:414] "containerMap: removing container" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803221 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803332 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803339 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="58507410-d70e-4a94-8e65-9861b2d7bf99" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803350 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803356 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacb3b25-4d18-484d-8aba-73b61f6a411e" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803366 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803373 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d81c1ad-ba94-4417-a066-ae7a1e74c2b9" containerName="maas-api" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803380 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7357f2-2289-4074-bf14-fe315ad33e0a" containerName="istio-proxy" Oct 30 21:54:03.803518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.803388 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f419dc-a8f9-473b-8b76-8fded5bbb494" containerName="maas-api" Oct 30 21:54:03.808552 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.808533 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:03.916827 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.916792 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e9aa8e-dockercfg-l4l2p\"" Oct 30 21:54:03.917093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.916908 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e9aa8e\"" Oct 30 21:54:03.933920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.932145 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2"] Oct 30 21:54:03.969908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.964342 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z652l\" (UniqueName: \"kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l\") pod \"payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:03.969908 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:03.964391 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:04.082010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.065673 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj"] Oct 30 21:54:04.082010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.071928 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z652l\" (UniqueName: \"kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l\") pod \"payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:04.082010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.071981 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:04.082010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.072968 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:04.082010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.076095 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.082010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.076916 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj"] Oct 30 21:54:04.131575 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.131536 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z652l\" (UniqueName: \"kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l\") pod \"payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:04.173388 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.173350 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.173388 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.173387 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csld6\" (UniqueName: \"kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6\") pod \"payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.274953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.274915 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.275162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.274960 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csld6\" (UniqueName: \"kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6\") pod \"payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.275558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.275534 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.332350 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.332280 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csld6\" (UniqueName: \"kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6\") pod \"payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.398725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.398693 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:04.422941 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.422911 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:04.784737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.784715 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj"] Oct 30 21:54:04.785276 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:54:04.785248 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a80e15_dc93_46fd_ad5d_7bcbb1302717.slice/crio-07cf77b2d48a26339edaed015ece290ef42dd53ecdd3a45f1c012b74f2e61580 WatchSource:0}: Error finding container 07cf77b2d48a26339edaed015ece290ef42dd53ecdd3a45f1c012b74f2e61580: Status 404 returned error can't find the container with id 07cf77b2d48a26339edaed015ece290ef42dd53ecdd3a45f1c012b74f2e61580 Oct 30 21:54:04.833763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:04.833738 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2"] Oct 30 21:54:04.848314 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:54:04.848289 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c06fe3d_e407_4403_936d_ece9e1e72116.slice/crio-252bf84c762bf4ed06d4ae98cf5b25d4e875b6f08755313372e66b87e574dafb WatchSource:0}: Error finding container 252bf84c762bf4ed06d4ae98cf5b25d4e875b6f08755313372e66b87e574dafb: Status 404 returned error can't find the container with id 252bf84c762bf4ed06d4ae98cf5b25d4e875b6f08755313372e66b87e574dafb Oct 30 21:54:05.775871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.775829 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" event={"ID":"48a80e15-dc93-46fd-ad5d-7bcbb1302717","Type":"ContainerStarted","Data":"44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392"} Oct 30 21:54:05.775871 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.775907 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:05.776167 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.775931 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" event={"ID":"48a80e15-dc93-46fd-ad5d-7bcbb1302717","Type":"ContainerStarted","Data":"07cf77b2d48a26339edaed015ece290ef42dd53ecdd3a45f1c012b74f2e61580"} Oct 30 21:54:05.777531 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.777492 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" event={"ID":"9c06fe3d-e407-4403-936d-ece9e1e72116","Type":"ContainerStarted","Data":"44b9215d4fcc5ab5c3be6fd207d77124432ac54f9c1bb9161dd3bc35e5d861ca"} Oct 30 21:54:05.777656 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.777552 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" event={"ID":"9c06fe3d-e407-4403-936d-ece9e1e72116","Type":"ContainerStarted","Data":"252bf84c762bf4ed06d4ae98cf5b25d4e875b6f08755313372e66b87e574dafb"} Oct 30 21:54:05.777656 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.777583 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:05.913004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:05.912948 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" podStartSLOduration=2.91293387 podStartE2EDuration="2.91293387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:05.910050711 +0000 UTC m=+2197.811914777" watchObservedRunningTime="2026-10-30 21:54:05.91293387 +0000 UTC m=+2197.814797934" Oct 30 21:54:07.018312 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:07.018259 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" podStartSLOduration=4.018244733 podStartE2EDuration="4.018244733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:06.05803218 +0000 UTC m=+2197.959896245" watchObservedRunningTime="2026-10-30 21:54:07.018244733 +0000 UTC m=+2198.920108798" Oct 30 21:54:07.020367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:07.020343 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb"] Oct 30 21:54:07.068703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:07.068671 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx"] Oct 30 21:54:12.346303 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:12.346259 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:54:12.630304 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:12.630270 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq"] Oct 30 21:54:12.630535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:12.630512 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" containerID="cri-o://425783994b5b50b7df7db3754400315f3a613a90573c04bb387b5f21242f51e3" gracePeriod=30 Oct 30 21:54:13.636996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:13.636960 5914 patch_prober.go:29] interesting pod/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:54:13.637456 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:13.637026 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:54:16.792445 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:16.792414 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 21:54:16.797423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:16.797392 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 21:54:17.880278 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.880239 5914 generic.go:356] "Generic (PLEG): container finished" podID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerID="425783994b5b50b7df7db3754400315f3a613a90573c04bb387b5f21242f51e3" exitCode=0 Oct 30 21:54:17.880704 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.880305 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" event={"ID":"e39efa30-1507-4ebb-a488-1dd88d1a22a6","Type":"ContainerDied","Data":"425783994b5b50b7df7db3754400315f3a613a90573c04bb387b5f21242f51e3"} Oct 30 21:54:17.918345 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.918318 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:54:17.947492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947461 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947493 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947679 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947599 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947692 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947761 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947737 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947761 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947756 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947847 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947930 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.947993 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947953 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:17.948035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947962 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:17.948035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.947967 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket\") pod \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\" (UID: \"e39efa30-1507-4ebb-a488-1dd88d1a22a6\") " Oct 30 21:54:17.948108 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.948042 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:17.948201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.948176 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:17.948201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.948188 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:17.948979 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.948951 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:17.949118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.948982 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:17.949118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.948999 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:17.949118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.949013 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:17.949118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.949028 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:17.950158 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.950136 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "kube-api-access-8qmmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:17.950287 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.950270 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:17.950361 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.950342 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:17.950443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:17.950418 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo" pod "e39efa30-1507-4ebb-a488-1dd88d1a22a6" (UID: "e39efa30-1507-4ebb-a488-1dd88d1a22a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:18.050438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.050405 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmmz\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-kube-api-access-8qmmz\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:18.050438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.050437 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:18.050631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.050447 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:18.050631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.050455 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39efa30-1507-4ebb-a488-1dd88d1a22a6-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:18.889482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.889454 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" Oct 30 21:54:18.889482 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.889471 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq" event={"ID":"e39efa30-1507-4ebb-a488-1dd88d1a22a6","Type":"ContainerDied","Data":"852eb1c1da1bb7e218a804c237ad77a367910fd3d8f4da75ffaa8a855f85c08f"} Oct 30 21:54:18.890027 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:18.889548 5914 scope.go:122] "RemoveContainer" containerID="425783994b5b50b7df7db3754400315f3a613a90573c04bb387b5f21242f51e3" Oct 30 21:54:19.114789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:19.114746 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq"] Oct 30 21:54:19.119198 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:19.119167 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-021907-openshift-default-dfccf5bf7-dwvhq"] Oct 30 21:54:20.100292 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:20.100256 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj"] Oct 30 21:54:20.100711 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:20.100480 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" containerID="cri-o://4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81" gracePeriod=30 Oct 30 21:54:20.218992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:20.218958 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" path="/var/lib/kubelet/pods/e39efa30-1507-4ebb-a488-1dd88d1a22a6/volumes" Oct 30 21:54:22.047533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.047473 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" containerID="cri-o://59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef" gracePeriod=45 Oct 30 21:54:22.090342 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.090308 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" containerID="cri-o://649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b" gracePeriod=45 Oct 30 21:54:22.389825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.389802 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:54:22.411290 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.411264 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:54:22.514089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514054 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs\") pod \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " Oct 30 21:54:22.514263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514117 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache\") pod \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " Oct 30 21:54:22.514263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514154 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home\") pod \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " Oct 30 21:54:22.514263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514212 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home\") pod \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " Oct 30 21:54:22.514263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514254 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir\") pod \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " Oct 30 21:54:22.514490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514312 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t\" (UniqueName: \"kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t\") pod \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " Oct 30 21:54:22.514490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514339 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm\") pod \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " Oct 30 21:54:22.514490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514413 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn\" (UniqueName: \"kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn\") pod \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " Oct 30 21:54:22.514641 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514546 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache\") pod \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " Oct 30 21:54:22.514641 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514613 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm\") pod \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " Oct 30 21:54:22.514797 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514779 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs\") pod \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\" (UID: \"a0d2491f-fbab-4843-9793-17fbd81dcb8c\") " Oct 30 21:54:22.514920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.514864 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir\") pod \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\" (UID: \"9c4cee3d-3fd2-453a-b01b-0499c6d2a773\") " Oct 30 21:54:22.516714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.516683 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir" pod "9c4cee3d-3fd2-453a-b01b-0499c6d2a773" (UID: "9c4cee3d-3fd2-453a-b01b-0499c6d2a773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.517498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.517176 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home" pod "a0d2491f-fbab-4843-9793-17fbd81dcb8c" (UID: "a0d2491f-fbab-4843-9793-17fbd81dcb8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.517498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.517275 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache" pod "9c4cee3d-3fd2-453a-b01b-0499c6d2a773" (UID: "9c4cee3d-3fd2-453a-b01b-0499c6d2a773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.517498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.517294 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache" pod "a0d2491f-fbab-4843-9793-17fbd81dcb8c" (UID: "a0d2491f-fbab-4843-9793-17fbd81dcb8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.517498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.517488 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home" pod "9c4cee3d-3fd2-453a-b01b-0499c6d2a773" (UID: "9c4cee3d-3fd2-453a-b01b-0499c6d2a773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.517847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.517683 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir" pod "a0d2491f-fbab-4843-9793-17fbd81dcb8c" (UID: "a0d2491f-fbab-4843-9793-17fbd81dcb8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.520428 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.520402 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm" pod "9c4cee3d-3fd2-453a-b01b-0499c6d2a773" (UID: "9c4cee3d-3fd2-453a-b01b-0499c6d2a773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.520679 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.520657 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs" pod "9c4cee3d-3fd2-453a-b01b-0499c6d2a773" (UID: "9c4cee3d-3fd2-453a-b01b-0499c6d2a773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:22.520798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.520724 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm" pod "a0d2491f-fbab-4843-9793-17fbd81dcb8c" (UID: "a0d2491f-fbab-4843-9793-17fbd81dcb8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:22.520798 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.520691 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t" pod "9c4cee3d-3fd2-453a-b01b-0499c6d2a773" (UID: "9c4cee3d-3fd2-453a-b01b-0499c6d2a773"). InnerVolumeSpecName "kube-api-access-47v6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:22.521030 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.521004 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn" pod "a0d2491f-fbab-4843-9793-17fbd81dcb8c" (UID: "a0d2491f-fbab-4843-9793-17fbd81dcb8c"). InnerVolumeSpecName "kube-api-access-qvndn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:22.522084 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.522062 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs" pod "a0d2491f-fbab-4843-9793-17fbd81dcb8c" (UID: "a0d2491f-fbab-4843-9793-17fbd81dcb8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618237 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618264 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618274 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618282 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618290 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618303 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618315 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618326 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47v6t\" (UniqueName: \"kubernetes.io/projected/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-kube-api-access-47v6t\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618335 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618343 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvndn\" (UniqueName: \"kubernetes.io/projected/a0d2491f-fbab-4843-9793-17fbd81dcb8c-kube-api-access-qvndn\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618353 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2491f-fbab-4843-9793-17fbd81dcb8c-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.618632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.618361 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4cee3d-3fd2-453a-b01b-0499c6d2a773-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:22.921051 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.921015 5914 generic.go:356] "Generic (PLEG): container finished" podID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerID="59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef" exitCode=0 Oct 30 21:54:22.921241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.921067 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" Oct 30 21:54:22.921241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.921103 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" event={"ID":"a0d2491f-fbab-4843-9793-17fbd81dcb8c","Type":"ContainerDied","Data":"59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef"} Oct 30 21:54:22.921241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.921158 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb" event={"ID":"a0d2491f-fbab-4843-9793-17fbd81dcb8c","Type":"ContainerDied","Data":"9680047fdde2e0b7130c9e28b8f2f1af18fedce40af3cb16f5e0f9fda45079b3"} Oct 30 21:54:22.921241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.921183 5914 scope.go:122] "RemoveContainer" containerID="59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef" Oct 30 21:54:22.923143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.923123 5914 generic.go:356] "Generic (PLEG): container finished" podID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerID="649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b" exitCode=0 Oct 30 21:54:22.923258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.923178 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" Oct 30 21:54:22.923258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.923227 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" event={"ID":"9c4cee3d-3fd2-453a-b01b-0499c6d2a773","Type":"ContainerDied","Data":"649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b"} Oct 30 21:54:22.923378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.923268 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" event={"ID":"9c4cee3d-3fd2-453a-b01b-0499c6d2a773","Type":"ContainerDied","Data":"1e26845c0d3da45dcf726f0ac56bf80a7afc6e16f732d1905348b087bfe063ba"} Oct 30 21:54:22.933400 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.933383 5914 scope.go:122] "RemoveContainer" containerID="59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef" Oct 30 21:54:22.933791 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:22.933745 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef\": container with ID starting with 59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef not found: ID does not exist" containerID="59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef" Oct 30 21:54:22.933791 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.933773 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef"} err="failed to get container status \"59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef\": rpc error: code = NotFound desc = could not find container \"59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef\": container with ID starting with 59ad8f48798345b11901d5bd4a2fa302f29b9fc1ddf56248d40a6d16f8cd9eef not found: ID does not exist" Oct 30 21:54:22.933931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.933794 5914 scope.go:122] "RemoveContainer" containerID="649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b" Oct 30 21:54:22.943188 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.943165 5914 scope.go:122] "RemoveContainer" containerID="649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b" Oct 30 21:54:22.943493 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:22.943472 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b\": container with ID starting with 649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b not found: ID does not exist" containerID="649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b" Oct 30 21:54:22.943575 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:22.943501 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b"} err="failed to get container status \"649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b\": rpc error: code = NotFound desc = could not find container \"649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b\": container with ID starting with 649d00525a9c2d513e9fa1e7260d34998e166908bb51b1380af8fe3c0b4e807b not found: ID does not exist" Oct 30 21:54:23.095590 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.095558 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb"] Oct 30 21:54:23.120870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.120842 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-2-w6-89c0fd-kserve-df7bcf574-pftpb"] Oct 30 21:54:23.212926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.212834 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw"] Oct 30 21:54:23.232001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.231968 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx"] Oct 30 21:54:23.243268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.243242 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx"] Oct 30 21:54:23.320208 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.320171 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj"] Oct 30 21:54:23.377685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:23.377637 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-89c0fd/distinct-w6-89c0fd-kserve-6d9c495fdf-qxllx" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:24.220217 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:24.220184 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" path="/var/lib/kubelet/pods/9c4cee3d-3fd2-453a-b01b-0499c6d2a773/volumes" Oct 30 21:54:24.220718 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:24.220701 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" path="/var/lib/kubelet/pods/a0d2491f-fbab-4843-9793-17fbd81dcb8c/volumes" Oct 30 21:54:25.380620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.380596 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:54:25.516954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.516922 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.516994 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517102 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517157 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517188 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517348 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517296 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:25.517553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517309 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517348 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:25.517553 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517541 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:25.517699 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517610 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517699 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517664 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517707 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket\") pod \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\" (UID: \"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e\") " Oct 30 21:54:25.517996 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.517851 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:25.518376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.518285 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:25.518376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.518326 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.518376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.518342 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.518376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.518351 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.518376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.518360 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.519575 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.519555 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:25.520070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.520042 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:25.520172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.520110 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "kube-api-access-mmvfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:25.520172 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.520144 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo" pod "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" (UID: "06f3aaea-fd91-4970-8bb6-2bc08ecdf22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:25.620238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.620204 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.620238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.620235 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.620238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.620244 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.620473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.620253 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.620473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.620264 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvfx\" (UniqueName: \"kubernetes.io/projected/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e-kube-api-access-mmvfx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:25.955916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.955853 5914 generic.go:356] "Generic (PLEG): container finished" podID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerID="4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81" exitCode=0 Oct 30 21:54:25.956129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.955933 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" Oct 30 21:54:25.956129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.955944 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" event={"ID":"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e","Type":"ContainerDied","Data":"4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81"} Oct 30 21:54:25.956129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.956013 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj" event={"ID":"06f3aaea-fd91-4970-8bb6-2bc08ecdf22e","Type":"ContainerDied","Data":"c5cfa124107d8e9528b476e7168ee8b3877810ed4cfcc17e443797a5af828e36"} Oct 30 21:54:25.956129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.956042 5914 scope.go:122] "RemoveContainer" containerID="4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81" Oct 30 21:54:25.967609 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.967591 5914 scope.go:122] "RemoveContainer" containerID="4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81" Oct 30 21:54:25.967922 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:25.967871 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81\": container with ID starting with 4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81 not found: ID does not exist" containerID="4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81" Oct 30 21:54:25.967976 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:25.967933 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81"} err="failed to get container status \"4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81\": rpc error: code = NotFound desc = could not find container \"4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81\": container with ID starting with 4d30dc7227cacd5473d8c139ac26a7a81946ed741d016570ac4d1d5ce5f0ff81 not found: ID does not exist" Oct 30 21:54:26.127817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:26.127781 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj"] Oct 30 21:54:26.177048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:26.177013 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-qxnxj"] Oct 30 21:54:26.219097 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:26.219021 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" path="/var/lib/kubelet/pods/06f3aaea-fd91-4970-8bb6-2bc08ecdf22e/volumes" Oct 30 21:54:28.213244 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.213188 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" podUID="ed6110a5-3d09-4d18-a591-be31f5c5e1d4" containerName="payload-processing" containerID="cri-o://d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5" gracePeriod=25 Oct 30 21:54:28.321300 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.321255 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" podUID="d13b974f-3697-4db8-85df-047cc6c7aab7" containerName="payload-pre-processing" containerID="cri-o://0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218" gracePeriod=25 Oct 30 21:54:28.512456 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.512404 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:54:28.560668 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.560635 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume\") pod \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " Oct 30 21:54:28.560863 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.560704 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf\" (UniqueName: \"kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf\") pod \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\" (UID: \"ed6110a5-3d09-4d18-a591-be31f5c5e1d4\") " Oct 30 21:54:28.561086 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.561061 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume" pod "ed6110a5-3d09-4d18-a591-be31f5c5e1d4" (UID: "ed6110a5-3d09-4d18-a591-be31f5c5e1d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:28.561306 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.561287 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.563933 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.563903 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf" pod "ed6110a5-3d09-4d18-a591-be31f5c5e1d4" (UID: "ed6110a5-3d09-4d18-a591-be31f5c5e1d4"). InnerVolumeSpecName "kube-api-access-f5vtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:28.602214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.602187 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:54:28.662349 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.662320 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9\" (UniqueName: \"kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9\") pod \"d13b974f-3697-4db8-85df-047cc6c7aab7\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " Oct 30 21:54:28.662532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.662514 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume\") pod \"d13b974f-3697-4db8-85df-047cc6c7aab7\" (UID: \"d13b974f-3697-4db8-85df-047cc6c7aab7\") " Oct 30 21:54:28.662872 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.662847 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume" pod "d13b974f-3697-4db8-85df-047cc6c7aab7" (UID: "d13b974f-3697-4db8-85df-047cc6c7aab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:28.662993 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.662939 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vtf\" (UniqueName: \"kubernetes.io/projected/ed6110a5-3d09-4d18-a591-be31f5c5e1d4-kube-api-access-f5vtf\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.664635 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.664615 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9" pod "d13b974f-3697-4db8-85df-047cc6c7aab7" (UID: "d13b974f-3697-4db8-85df-047cc6c7aab7"). InnerVolumeSpecName "kube-api-access-n72w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:28.763961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.763847 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13b974f-3697-4db8-85df-047cc6c7aab7-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.763961 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.763893 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72w9\" (UniqueName: \"kubernetes.io/projected/d13b974f-3697-4db8-85df-047cc6c7aab7-kube-api-access-n72w9\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:28.983222 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.983179 5914 generic.go:356] "Generic (PLEG): container finished" podID="ed6110a5-3d09-4d18-a591-be31f5c5e1d4" containerID="d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5" exitCode=0 Oct 30 21:54:28.983413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.983221 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" event={"ID":"ed6110a5-3d09-4d18-a591-be31f5c5e1d4","Type":"ContainerDied","Data":"d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5"} Oct 30 21:54:28.983413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.983257 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" Oct 30 21:54:28.983413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.983305 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw" event={"ID":"ed6110a5-3d09-4d18-a591-be31f5c5e1d4","Type":"ContainerDied","Data":"48ce95c4fd880e1dc2614d9a5668b03b1da17dc984ba246a41aee981e52da5d3"} Oct 30 21:54:28.983413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.983339 5914 scope.go:122] "RemoveContainer" containerID="d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5" Oct 30 21:54:28.985826 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.985799 5914 generic.go:356] "Generic (PLEG): container finished" podID="d13b974f-3697-4db8-85df-047cc6c7aab7" containerID="0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218" exitCode=0 Oct 30 21:54:28.985985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.985862 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" Oct 30 21:54:28.985985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.985901 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" event={"ID":"d13b974f-3697-4db8-85df-047cc6c7aab7","Type":"ContainerDied","Data":"0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218"} Oct 30 21:54:28.985985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.985957 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj" event={"ID":"d13b974f-3697-4db8-85df-047cc6c7aab7","Type":"ContainerDied","Data":"e6dc970fcd2d055b70b0c71e80194e437fd47694f6ca3cc9c2058a8b2efd8bd4"} Oct 30 21:54:28.996856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.996830 5914 scope.go:122] "RemoveContainer" containerID="d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5" Oct 30 21:54:28.997210 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:28.997189 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5\": container with ID starting with d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5 not found: ID does not exist" containerID="d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5" Oct 30 21:54:28.997294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.997218 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5"} err="failed to get container status \"d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5\": rpc error: code = NotFound desc = could not find container \"d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5\": container with ID starting with d15b74767cafa55cfb956eb6b59e9d726506917439d7a4163409940185a51df5 not found: ID does not exist" Oct 30 21:54:28.997294 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:28.997237 5914 scope.go:122] "RemoveContainer" containerID="0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218" Oct 30 21:54:29.008429 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:29.008411 5914 scope.go:122] "RemoveContainer" containerID="0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218" Oct 30 21:54:29.008734 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:29.008712 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218\": container with ID starting with 0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218 not found: ID does not exist" containerID="0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218" Oct 30 21:54:29.008825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:29.008741 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218"} err="failed to get container status \"0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218\": rpc error: code = NotFound desc = could not find container \"0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218\": container with ID starting with 0c324fb1f7424be3a16c57a570e1fd6a7924c389223c469971d631ceb00dc218 not found: ID does not exist" Oct 30 21:54:29.153257 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:29.153226 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw"] Oct 30 21:54:29.202390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:29.202353 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9c76ae-5778b64c47-9sqrw"] Oct 30 21:54:29.301336 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:29.301307 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj"] Oct 30 21:54:29.326413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:29.326381 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-9c76ae-685979f455-kbhsj"] Oct 30 21:54:30.219444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:30.219409 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13b974f-3697-4db8-85df-047cc6c7aab7" path="/var/lib/kubelet/pods/d13b974f-3697-4db8-85df-047cc6c7aab7/volumes" Oct 30 21:54:30.219875 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:30.219859 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6110a5-3d09-4d18-a591-be31f5c5e1d4" path="/var/lib/kubelet/pods/ed6110a5-3d09-4d18-a591-be31f5c5e1d4/volumes" Oct 30 21:54:31.005179 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.005142 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j"] Oct 30 21:54:31.005603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.005314 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" podUID="59bbd2ec-0a1e-406a-945a-a271d0e66bc5" containerName="maas-api" containerID="cri-o://70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125" gracePeriod=30 Oct 30 21:54:31.353557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.353522 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:54:31.423215 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.423183 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\") pod \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " Oct 30 21:54:31.423413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.423238 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs\") pod \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " Oct 30 21:54:31.423413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.423317 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw\" (UniqueName: \"kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw\") pod \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " Oct 30 21:54:31.423413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.423337 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca\") pod \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\" (UID: \"59bbd2ec-0a1e-406a-945a-a271d0e66bc5\") " Oct 30 21:54:31.423818 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.423787 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca" pod "59bbd2ec-0a1e-406a-945a-a271d0e66bc5" (UID: "59bbd2ec-0a1e-406a-945a-a271d0e66bc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:31.425555 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.425520 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs" pod "59bbd2ec-0a1e-406a-945a-a271d0e66bc5" (UID: "59bbd2ec-0a1e-406a-945a-a271d0e66bc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:31.426049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.426031 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw" pod "59bbd2ec-0a1e-406a-945a-a271d0e66bc5" (UID: "59bbd2ec-0a1e-406a-945a-a271d0e66bc5"). InnerVolumeSpecName "kube-api-access-k84cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:31.426126 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.426048 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls" pod "59bbd2ec-0a1e-406a-945a-a271d0e66bc5" (UID: "59bbd2ec-0a1e-406a-945a-a271d0e66bc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:31.529974 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.527124 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84cw\" (UniqueName: \"kubernetes.io/projected/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-kube-api-access-k84cw\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.530228 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.530205 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.530319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.530305 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:31.530401 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:31.530390 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59bbd2ec-0a1e-406a-945a-a271d0e66bc5-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.008121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.008081 5914 generic.go:356] "Generic (PLEG): container finished" podID="59bbd2ec-0a1e-406a-945a-a271d0e66bc5" containerID="70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125" exitCode=0 Oct 30 21:54:32.008587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.008136 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" Oct 30 21:54:32.008587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.008135 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" event={"ID":"59bbd2ec-0a1e-406a-945a-a271d0e66bc5","Type":"ContainerDied","Data":"70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125"} Oct 30 21:54:32.008587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.008250 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j" event={"ID":"59bbd2ec-0a1e-406a-945a-a271d0e66bc5","Type":"ContainerDied","Data":"0e7b1c74667ac999c67460e52c0cae2dc91c96bd952f4fbb0ab4f1977cc9e277"} Oct 30 21:54:32.008587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.008279 5914 scope.go:122] "RemoveContainer" containerID="70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125" Oct 30 21:54:32.021122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.021071 5914 scope.go:122] "RemoveContainer" containerID="70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125" Oct 30 21:54:32.021416 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:32.021393 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125\": container with ID starting with 70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125 not found: ID does not exist" containerID="70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125" Oct 30 21:54:32.021469 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.021424 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125"} err="failed to get container status \"70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125\": rpc error: code = NotFound desc = could not find container \"70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125\": container with ID starting with 70b6b174f10107e6dbae810f302d6af467427af4dce61833f3317fc952763125 not found: ID does not exist" Oct 30 21:54:32.221627 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.221571 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j"] Oct 30 21:54:32.221831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.221642 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9c76ae-6cf7c787dc-2rk6j"] Oct 30 21:54:32.250424 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:32.249705 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bbd2ec-0a1e-406a-945a-a271d0e66bc5" path="/var/lib/kubelet/pods/59bbd2ec-0a1e-406a-945a-a271d0e66bc5/volumes" Oct 30 21:54:40.819899 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:40.819639 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj"] Oct 30 21:54:40.820408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:40.819961 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" containerID="cri-o://e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13" gracePeriod=30 Oct 30 21:54:41.098760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:41.098670 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:54:42.346944 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:42.346909 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:54:46.110824 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.110798 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:54:46.157083 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.157041 5914 generic.go:356] "Generic (PLEG): container finished" podID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerID="e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13" exitCode=0 Oct 30 21:54:46.157269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.157102 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" Oct 30 21:54:46.157269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.157123 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" event={"ID":"86b8a9d0-0196-4020-b114-73f3ffa1e7a9","Type":"ContainerDied","Data":"e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13"} Oct 30 21:54:46.157269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.157185 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj" event={"ID":"86b8a9d0-0196-4020-b114-73f3ffa1e7a9","Type":"ContainerDied","Data":"68a7417abe6cd4c5d8aa7132bb18697a54698def1c05b897549d3df5379daba8"} Oct 30 21:54:46.157269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.157217 5914 scope.go:122] "RemoveContainer" containerID="e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13" Oct 30 21:54:46.168068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.168039 5914 scope.go:122] "RemoveContainer" containerID="385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208" Oct 30 21:54:46.178763 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.178739 5914 scope.go:122] "RemoveContainer" containerID="e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13" Oct 30 21:54:46.179154 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:46.179130 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13\": container with ID starting with e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13 not found: ID does not exist" containerID="e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13" Oct 30 21:54:46.179213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.179164 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13"} err="failed to get container status \"e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13\": rpc error: code = NotFound desc = could not find container \"e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13\": container with ID starting with e202d0e44c95f604273f4e15054801a81f08e47c247bcb5936a3f3bb64bdba13 not found: ID does not exist" Oct 30 21:54:46.179213 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.179184 5914 scope.go:122] "RemoveContainer" containerID="385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208" Oct 30 21:54:46.179469 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:54:46.179446 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208\": container with ID starting with 385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208 not found: ID does not exist" containerID="385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208" Oct 30 21:54:46.179533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.179479 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208"} err="failed to get container status \"385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208\": rpc error: code = NotFound desc = could not find container \"385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208\": container with ID starting with 385050ebd25a3df6399babff4c6398dbb5361655629b02fb7e5751b726464208 not found: ID does not exist" Oct 30 21:54:46.197203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197167 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197232 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197263 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197286 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197326 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197357 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197394 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197466 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197488 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:46.197670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197524 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs\") pod \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\" (UID: \"86b8a9d0-0196-4020-b114-73f3ffa1e7a9\") " Oct 30 21:54:46.197907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197670 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:46.197907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197840 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:46.197907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197869 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:46.198057 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.197983 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:46.198248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.198230 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.198338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.198250 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.198338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.198261 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.198338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.198274 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.198338 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.198288 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.199745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.199724 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:46.199909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.199752 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:46.199965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.199934 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:46.200283 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.200264 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t" pod "86b8a9d0-0196-4020-b114-73f3ffa1e7a9" (UID: "86b8a9d0-0196-4020-b114-73f3ffa1e7a9"). InnerVolumeSpecName "kube-api-access-dq65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:46.300628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.300583 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.300628 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.300630 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq65t\" (UniqueName: \"kubernetes.io/projected/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-kube-api-access-dq65t\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.300917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.300647 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.300917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.300661 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b8a9d0-0196-4020-b114-73f3ffa1e7a9-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:46.576012 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.575975 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj"] Oct 30 21:54:46.579711 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:46.579684 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9c76ae-openshift-default-6f5b49f64f-k4fcj"] Oct 30 21:54:48.220017 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:48.219976 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" path="/var/lib/kubelet/pods/86b8a9d0-0196-4020-b114-73f3ffa1e7a9/volumes" Oct 30 21:54:49.872734 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:49.872698 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz"] Oct 30 21:54:49.873181 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:49.872983 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" podUID="982d2a1c-eacf-41cf-b9d7-649838190b31" containerName="maas-api" containerID="cri-o://598e132857a810869eb9e8e7889f6c7b3e39efc430220a370a16403564cf9b3a" gracePeriod=30 Oct 30 21:54:50.189489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.189459 5914 generic.go:356] "Generic (PLEG): container finished" podID="982d2a1c-eacf-41cf-b9d7-649838190b31" containerID="598e132857a810869eb9e8e7889f6c7b3e39efc430220a370a16403564cf9b3a" exitCode=0 Oct 30 21:54:50.189638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.189533 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" event={"ID":"982d2a1c-eacf-41cf-b9d7-649838190b31","Type":"ContainerDied","Data":"598e132857a810869eb9e8e7889f6c7b3e39efc430220a370a16403564cf9b3a"} Oct 30 21:54:50.217113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.217085 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:54:50.258261 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.258230 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs\") pod \"982d2a1c-eacf-41cf-b9d7-649838190b31\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " Oct 30 21:54:50.258449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.258281 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc\" (UniqueName: \"kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc\") pod \"982d2a1c-eacf-41cf-b9d7-649838190b31\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " Oct 30 21:54:50.258449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.258316 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\") pod \"982d2a1c-eacf-41cf-b9d7-649838190b31\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " Oct 30 21:54:50.258449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.258352 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca\") pod \"982d2a1c-eacf-41cf-b9d7-649838190b31\" (UID: \"982d2a1c-eacf-41cf-b9d7-649838190b31\") " Oct 30 21:54:50.259408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.258781 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca" pod "982d2a1c-eacf-41cf-b9d7-649838190b31" (UID: "982d2a1c-eacf-41cf-b9d7-649838190b31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:50.259408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.259309 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/982d2a1c-eacf-41cf-b9d7-649838190b31-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:50.260630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.260605 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs" pod "982d2a1c-eacf-41cf-b9d7-649838190b31" (UID: "982d2a1c-eacf-41cf-b9d7-649838190b31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:50.260745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.260712 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls" pod "982d2a1c-eacf-41cf-b9d7-649838190b31" (UID: "982d2a1c-eacf-41cf-b9d7-649838190b31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:50.260854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.260838 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc" pod "982d2a1c-eacf-41cf-b9d7-649838190b31" (UID: "982d2a1c-eacf-41cf-b9d7-649838190b31"). InnerVolumeSpecName "kube-api-access-xwxpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:50.363319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.362792 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:50.363319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.362829 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwxpc\" (UniqueName: \"kubernetes.io/projected/982d2a1c-eacf-41cf-b9d7-649838190b31-kube-api-access-xwxpc\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:50.363319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:50.362847 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/982d2a1c-eacf-41cf-b9d7-649838190b31-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:54:51.197982 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:51.197949 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" Oct 30 21:54:51.198421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:51.197948 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz" event={"ID":"982d2a1c-eacf-41cf-b9d7-649838190b31","Type":"ContainerDied","Data":"fe3fd48c0105f959ae9298768d5d16db4d0f75dd217d60751c36754c991e2c4e"} Oct 30 21:54:51.198421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:51.198112 5914 scope.go:122] "RemoveContainer" containerID="598e132857a810869eb9e8e7889f6c7b3e39efc430220a370a16403564cf9b3a" Oct 30 21:54:51.411411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:51.411380 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz"] Oct 30 21:54:51.460337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:51.460264 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-013fb2-6c4c959c87-7czgz"] Oct 30 21:54:52.217967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:54:52.217863 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982d2a1c-eacf-41cf-b9d7-649838190b31" path="/var/lib/kubelet/pods/982d2a1c-eacf-41cf-b9d7-649838190b31/volumes" Oct 30 21:55:12.346277 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:55:12.346186 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:55:13.227760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.227725 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6"] Oct 30 21:55:13.228052 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.228021 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" containerID="cri-o://ca769f274d18fec1e8bfbeaeb1cbe0dde89fb18c5ed6f7eaf0c2e909276ea8c1" gracePeriod=30 Oct 30 21:55:13.271796 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.271758 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh"] Oct 30 21:55:13.272039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.271974 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" containerID="cri-o://0d730ef5c4c889cdd5ef8e3eaec5d035bab931772717db13f39f0f001da32aa3" gracePeriod=30 Oct 30 21:55:13.274914 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.274867 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2"] Oct 30 21:55:13.275139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.275112 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" containerID="cri-o://349b91f82e9459522ca27ebb2b44773b0797aa1ba9ae3edd0e07eb8e56e607a1" gracePeriod=30 Oct 30 21:55:13.321452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.321412 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2"] Oct 30 21:55:13.321693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:13.321666 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" containerID="cri-o://a6dd04de3eb79031fe011941e8ed667b0fa5b6c9054ec899b449aa224add5edf" gracePeriod=30 Oct 30 21:55:17.374238 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:17.374196 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-qjkt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:55:17.374733 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:17.374250 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:55:17.466112 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:17.466079 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lbxq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:55:17.466308 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:17.466145 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:55:18.516357 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.516252 5914 generic.go:356] "Generic (PLEG): container finished" podID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerID="349b91f82e9459522ca27ebb2b44773b0797aa1ba9ae3edd0e07eb8e56e607a1" exitCode=0 Oct 30 21:55:18.516766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.516416 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" event={"ID":"3e554433-27a6-4edb-bc0e-416f50b6bb87","Type":"ContainerDied","Data":"349b91f82e9459522ca27ebb2b44773b0797aa1ba9ae3edd0e07eb8e56e607a1"} Oct 30 21:55:18.520265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.520227 5914 generic.go:356] "Generic (PLEG): container finished" podID="5b30a216-a370-463e-b9c9-125427eec2fb" containerID="0d730ef5c4c889cdd5ef8e3eaec5d035bab931772717db13f39f0f001da32aa3" exitCode=0 Oct 30 21:55:18.520437 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.520366 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" event={"ID":"5b30a216-a370-463e-b9c9-125427eec2fb","Type":"ContainerDied","Data":"0d730ef5c4c889cdd5ef8e3eaec5d035bab931772717db13f39f0f001da32aa3"} Oct 30 21:55:18.522789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.522742 5914 generic.go:356] "Generic (PLEG): container finished" podID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerID="a6dd04de3eb79031fe011941e8ed667b0fa5b6c9054ec899b449aa224add5edf" exitCode=0 Oct 30 21:55:18.522975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.522954 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" event={"ID":"5fad0ca5-639a-474f-aa48-51184c8adbf8","Type":"ContainerDied","Data":"a6dd04de3eb79031fe011941e8ed667b0fa5b6c9054ec899b449aa224add5edf"} Oct 30 21:55:18.524952 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.524926 5914 generic.go:356] "Generic (PLEG): container finished" podID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerID="ca769f274d18fec1e8bfbeaeb1cbe0dde89fb18c5ed6f7eaf0c2e909276ea8c1" exitCode=0 Oct 30 21:55:18.525071 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.525001 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" event={"ID":"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9","Type":"ContainerDied","Data":"ca769f274d18fec1e8bfbeaeb1cbe0dde89fb18c5ed6f7eaf0c2e909276ea8c1"} Oct 30 21:55:18.529913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.529562 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-bc7r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:55:18.529913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.529630 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 21:55:18.601637 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.601609 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:55:18.684822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.684758 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.684831 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.684924 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.685038 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.685080 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.685194 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.685249 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685309 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.685301 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.685396 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.685334 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t\") pod \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\" (UID: \"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9\") " Oct 30 21:55:18.686471 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.686434 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.686683 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.686660 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.686920 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.686897 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.691367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.691340 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.691542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.691488 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:18.691756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.691730 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.693653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.693236 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:55:18.694991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.694941 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:18.700541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.700474 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "kube-api-access-qdz5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.716844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.716803 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy" pod "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" (UID: "6369fbb0-e0ba-43b4-92ec-c8f2afae70b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.739951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.739919 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:55:18.743445 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.743420 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:55:18.787535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787444 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.787535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787486 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.787772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787575 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.787772 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787646 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.787913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787793 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.787913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787832 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.787913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787907 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.788083 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787948 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.788083 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.787987 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.788083 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788013 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:18.788243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788090 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy\") pod \"3e554433-27a6-4edb-bc0e-416f50b6bb87\" (UID: \"3e554433-27a6-4edb-bc0e-416f50b6bb87\") " Oct 30 21:55:18.788243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788215 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.788901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788433 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.788901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788485 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.788901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788855 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.788901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788898 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788915 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788931 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788947 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdz5t\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-kube-api-access-qdz5t\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788963 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788977 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.788991 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.789004 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.789017 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.789030 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.789045 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e554433-27a6-4edb-bc0e-416f50b6bb87-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.789058 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.789207 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.789071 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.790518 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.790491 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.790613 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.790540 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "kube-api-access-24g74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.790765 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.790751 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:18.790932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.790916 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy" pod "3e554433-27a6-4edb-bc0e-416f50b6bb87" (UID: "3e554433-27a6-4edb-bc0e-416f50b6bb87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.890440 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890406 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.890631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890465 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.890631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890482 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.890631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890502 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.890631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890542 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.890975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890734 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.890975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890838 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.890975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890914 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.891158 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891100 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.891158 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891139 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.891272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.890865 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.891272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891039 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:18.891272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891200 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.891272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891268 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.891467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891314 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.891467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891132 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.891467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891309 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.891467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891347 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.891467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891434 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.891467 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891465 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.891785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891508 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.891785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891576 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.891785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891610 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo\") pod \"5b30a216-a370-463e-b9c9-125427eec2fb\" (UID: \"5b30a216-a370-463e-b9c9-125427eec2fb\") " Oct 30 21:55:18.891785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.891656 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token\") pod \"5fad0ca5-639a-474f-aa48-51184c8adbf8\" (UID: \"5fad0ca5-639a-474f-aa48-51184c8adbf8\") " Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892251 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892417 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892549 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892570 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g74\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-kube-api-access-24g74\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892586 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892599 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892613 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892626 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892638 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b30a216-a370-463e-b9c9-125427eec2fb-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892651 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892663 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.892673 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892675 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e554433-27a6-4edb-bc0e-416f50b6bb87-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.893416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892688 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.893416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.892701 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.893416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.893051 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.893568 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.893446 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:18.893749 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.893726 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.894122 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.894092 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.894595 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.894570 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.896714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.896688 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:18.896828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.896815 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "kube-api-access-w6vgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.896915 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.896860 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:18.899232 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.899209 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx" pod "5b30a216-a370-463e-b9c9-125427eec2fb" (UID: "5b30a216-a370-463e-b9c9-125427eec2fb"). InnerVolumeSpecName "kube-api-access-mbpdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:18.900861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.900838 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo" pod "5fad0ca5-639a-474f-aa48-51184c8adbf8" (UID: "5fad0ca5-639a-474f-aa48-51184c8adbf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:18.994479 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994442 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994479 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994479 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b30a216-a370-463e-b9c9-125427eec2fb-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994492 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994505 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vgk\" (UniqueName: \"kubernetes.io/projected/5fad0ca5-639a-474f-aa48-51184c8adbf8-kube-api-access-w6vgk\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994519 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b30a216-a370-463e-b9c9-125427eec2fb-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994531 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994543 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994555 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fad0ca5-639a-474f-aa48-51184c8adbf8-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994569 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbpdx\" (UniqueName: \"kubernetes.io/projected/5b30a216-a370-463e-b9c9-125427eec2fb-kube-api-access-mbpdx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:18.994695 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:18.994580 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fad0ca5-639a-474f-aa48-51184c8adbf8-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:19.532751 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.532666 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" Oct 30 21:55:19.532751 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.532685 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2" event={"ID":"3e554433-27a6-4edb-bc0e-416f50b6bb87","Type":"ContainerDied","Data":"1a8a6e4e209edbed54688535cb00d023564bf0568ccfb94c62335e59271897fa"} Oct 30 21:55:19.533314 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.532759 5914 scope.go:122] "RemoveContainer" containerID="349b91f82e9459522ca27ebb2b44773b0797aa1ba9ae3edd0e07eb8e56e607a1" Oct 30 21:55:19.534515 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.534493 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" Oct 30 21:55:19.534622 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.534518 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh" event={"ID":"5b30a216-a370-463e-b9c9-125427eec2fb","Type":"ContainerDied","Data":"4ce0d8589ec3dea161894111e342695614744602e2a1dccc5a34eefc59017172"} Oct 30 21:55:19.536470 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.536435 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" Oct 30 21:55:19.536595 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.536463 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2" event={"ID":"5fad0ca5-639a-474f-aa48-51184c8adbf8","Type":"ContainerDied","Data":"510d0a1ccbd1b6991c323ccd42dff118201a8ff305f6035a40957ed43c61d954"} Oct 30 21:55:19.538091 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.538067 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" event={"ID":"6369fbb0-e0ba-43b4-92ec-c8f2afae70b9","Type":"ContainerDied","Data":"edf8106f820bd83d0111a08706b3e68e530d746d761f15df1bb83cfe3da3baea"} Oct 30 21:55:19.538214 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.538125 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6" Oct 30 21:55:19.548103 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.548000 5914 scope.go:122] "RemoveContainer" containerID="0d730ef5c4c889cdd5ef8e3eaec5d035bab931772717db13f39f0f001da32aa3" Oct 30 21:55:19.561332 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.561312 5914 scope.go:122] "RemoveContainer" containerID="a6dd04de3eb79031fe011941e8ed667b0fa5b6c9054ec899b449aa224add5edf" Oct 30 21:55:19.570926 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.570905 5914 scope.go:122] "RemoveContainer" containerID="ca769f274d18fec1e8bfbeaeb1cbe0dde89fb18c5ed6f7eaf0c2e909276ea8c1" Oct 30 21:55:19.711640 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.711607 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh"] Oct 30 21:55:19.761134 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.761096 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-fzjdh"] Oct 30 21:55:19.841825 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.841733 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2"] Oct 30 21:55:19.845569 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.845541 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-bc7r2"] Oct 30 21:55:19.956064 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.956029 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6"] Oct 30 21:55:19.962519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:19.962491 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-qjkt6"] Oct 30 21:55:20.069704 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.069650 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2"] Oct 30 21:55:20.072657 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.072629 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lbxq2"] Oct 30 21:55:20.220847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.220811 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" path="/var/lib/kubelet/pods/3e554433-27a6-4edb-bc0e-416f50b6bb87/volumes" Oct 30 21:55:20.221557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.221523 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" path="/var/lib/kubelet/pods/5b30a216-a370-463e-b9c9-125427eec2fb/volumes" Oct 30 21:55:20.222242 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.222222 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" path="/var/lib/kubelet/pods/5fad0ca5-639a-474f-aa48-51184c8adbf8/volumes" Oct 30 21:55:20.223138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.223122 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" path="/var/lib/kubelet/pods/6369fbb0-e0ba-43b4-92ec-c8f2afae70b9/volumes" Oct 30 21:55:20.978730 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:20.977802 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q"] Oct 30 21:55:23.368620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:23.368584 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7"] Oct 30 21:55:23.475456 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:23.475424 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq"] Oct 30 21:55:25.979143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:25.979095 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" podUID="6f7c967b-3dad-4950-b77e-005dfb518f9c" containerName="payload-pre-processing" containerID="cri-o://462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f" gracePeriod=25 Oct 30 21:55:26.271402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.271374 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:55:26.376661 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.376624 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5\" (UniqueName: \"kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5\") pod \"6f7c967b-3dad-4950-b77e-005dfb518f9c\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " Oct 30 21:55:26.376866 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.376804 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume\") pod \"6f7c967b-3dad-4950-b77e-005dfb518f9c\" (UID: \"6f7c967b-3dad-4950-b77e-005dfb518f9c\") " Oct 30 21:55:26.377198 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.377173 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume" pod "6f7c967b-3dad-4950-b77e-005dfb518f9c" (UID: "6f7c967b-3dad-4950-b77e-005dfb518f9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:26.378981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.378950 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5" pod "6f7c967b-3dad-4950-b77e-005dfb518f9c" (UID: "6f7c967b-3dad-4950-b77e-005dfb518f9c"). InnerVolumeSpecName "kube-api-access-p4hb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:26.478200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.478158 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c967b-3dad-4950-b77e-005dfb518f9c-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:26.478200 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.478192 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4hb5\" (UniqueName: \"kubernetes.io/projected/6f7c967b-3dad-4950-b77e-005dfb518f9c-kube-api-access-p4hb5\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:26.615968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.615860 5914 generic.go:356] "Generic (PLEG): container finished" podID="6f7c967b-3dad-4950-b77e-005dfb518f9c" containerID="462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f" exitCode=0 Oct 30 21:55:26.615968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.615944 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" Oct 30 21:55:26.615968 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.615949 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" event={"ID":"6f7c967b-3dad-4950-b77e-005dfb518f9c","Type":"ContainerDied","Data":"462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f"} Oct 30 21:55:26.616229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.616013 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q" event={"ID":"6f7c967b-3dad-4950-b77e-005dfb518f9c","Type":"ContainerDied","Data":"74cf7236aea4e778aa49b1e372d12b863069d8b333e13d60200bd4f893305237"} Oct 30 21:55:26.616229 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.616038 5914 scope.go:122] "RemoveContainer" containerID="462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f" Oct 30 21:55:26.627046 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.627025 5914 scope.go:122] "RemoveContainer" containerID="462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f" Oct 30 21:55:26.627366 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:55:26.627345 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f\": container with ID starting with 462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f not found: ID does not exist" containerID="462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f" Oct 30 21:55:26.627450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.627375 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f"} err="failed to get container status \"462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f\": rpc error: code = NotFound desc = could not find container \"462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f\": container with ID starting with 462cb159d8335aa8d22d8ab3c7b4296c46c70fe45fbc7d175bd1d6c9d68d793f not found: ID does not exist" Oct 30 21:55:26.779785 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.779747 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q"] Oct 30 21:55:26.830739 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:26.830696 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ddec2-7446bd4979-j7h4q"] Oct 30 21:55:28.218293 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.218258 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7c967b-3dad-4950-b77e-005dfb518f9c" path="/var/lib/kubelet/pods/6f7c967b-3dad-4950-b77e-005dfb518f9c/volumes" Oct 30 21:55:28.370080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.370030 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" podUID="ef2ee665-ffed-422b-a220-6304d47f1800" containerName="payload-processing" containerID="cri-o://2dc83a232ca6e05acc20880a27b80086f4a486bfec14ad8404dcb97352dc85ea" gracePeriod=25 Oct 30 21:55:28.476501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.476397 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" podUID="5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" containerName="payload-pre-processing" containerID="cri-o://b6932319ddaf9e61ce370884e5b1d415f9947b75a0c8340fbeecb1a4c32620a4" gracePeriod=25 Oct 30 21:55:28.633074 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.633037 5914 generic.go:356] "Generic (PLEG): container finished" podID="5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" containerID="b6932319ddaf9e61ce370884e5b1d415f9947b75a0c8340fbeecb1a4c32620a4" exitCode=0 Oct 30 21:55:28.633257 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.633110 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" event={"ID":"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb","Type":"ContainerDied","Data":"b6932319ddaf9e61ce370884e5b1d415f9947b75a0c8340fbeecb1a4c32620a4"} Oct 30 21:55:28.634954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.634926 5914 generic.go:356] "Generic (PLEG): container finished" podID="ef2ee665-ffed-422b-a220-6304d47f1800" containerID="2dc83a232ca6e05acc20880a27b80086f4a486bfec14ad8404dcb97352dc85ea" exitCode=0 Oct 30 21:55:28.635117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.635006 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" event={"ID":"ef2ee665-ffed-422b-a220-6304d47f1800","Type":"ContainerDied","Data":"2dc83a232ca6e05acc20880a27b80086f4a486bfec14ad8404dcb97352dc85ea"} Oct 30 21:55:28.659591 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.659562 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:55:28.707111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.707073 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume\") pod \"ef2ee665-ffed-422b-a220-6304d47f1800\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " Oct 30 21:55:28.707301 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.707283 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns\" (UniqueName: \"kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns\") pod \"ef2ee665-ffed-422b-a220-6304d47f1800\" (UID: \"ef2ee665-ffed-422b-a220-6304d47f1800\") " Oct 30 21:55:28.707547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.707518 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume" pod "ef2ee665-ffed-422b-a220-6304d47f1800" (UID: "ef2ee665-ffed-422b-a220-6304d47f1800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:28.707904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.707854 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2ee665-ffed-422b-a220-6304d47f1800-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.709637 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.709606 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns" pod "ef2ee665-ffed-422b-a220-6304d47f1800" (UID: "ef2ee665-ffed-422b-a220-6304d47f1800"). InnerVolumeSpecName "kube-api-access-wgrns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:28.759674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.759649 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:55:28.808956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.808925 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume\") pod \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " Oct 30 21:55:28.809171 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.809020 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg\" (UniqueName: \"kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg\") pod \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\" (UID: \"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb\") " Oct 30 21:55:28.809443 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.809415 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume" pod "5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" (UID: "5c150d9e-7846-4ebd-82d0-2ed3790a4ecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:28.809757 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.809723 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.809757 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.809757 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgrns\" (UniqueName: \"kubernetes.io/projected/ef2ee665-ffed-422b-a220-6304d47f1800-kube-api-access-wgrns\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:28.811463 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.811436 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg" pod "5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" (UID: "5c150d9e-7846-4ebd-82d0-2ed3790a4ecb"). InnerVolumeSpecName "kube-api-access-dtbxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:28.910982 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:28.910949 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbxg\" (UniqueName: \"kubernetes.io/projected/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb-kube-api-access-dtbxg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:29.646347 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.646257 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" Oct 30 21:55:29.646796 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.646261 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq" event={"ID":"5c150d9e-7846-4ebd-82d0-2ed3790a4ecb","Type":"ContainerDied","Data":"9f93856e7f4fcc70d7284741a1ed69c4fda0675bcb4a2819cd53dccbf55f7d83"} Oct 30 21:55:29.646796 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.646410 5914 scope.go:122] "RemoveContainer" containerID="b6932319ddaf9e61ce370884e5b1d415f9947b75a0c8340fbeecb1a4c32620a4" Oct 30 21:55:29.648324 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.648297 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" event={"ID":"ef2ee665-ffed-422b-a220-6304d47f1800","Type":"ContainerDied","Data":"2a643d518845100c621cc96730adad624f51b874e9c78ea83c88301a1aa9c177"} Oct 30 21:55:29.648413 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.648397 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7" Oct 30 21:55:29.658615 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.658528 5914 scope.go:122] "RemoveContainer" containerID="2dc83a232ca6e05acc20880a27b80086f4a486bfec14ad8404dcb97352dc85ea" Oct 30 21:55:29.826101 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.826062 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7"] Oct 30 21:55:29.876710 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:29.876672 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1dc70e-866854d6d9-vf4f7"] Oct 30 21:55:30.014353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:30.012353 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq"] Oct 30 21:55:30.017058 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:30.017016 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1dc70e-759f98d859-jlbfq"] Oct 30 21:55:30.300576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:30.299913 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" path="/var/lib/kubelet/pods/5c150d9e-7846-4ebd-82d0-2ed3790a4ecb/volumes" Oct 30 21:55:30.300576 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:30.300556 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2ee665-ffed-422b-a220-6304d47f1800" path="/var/lib/kubelet/pods/ef2ee665-ffed-422b-a220-6304d47f1800/volumes" Oct 30 21:55:42.345766 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:55:42.345719 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:55:47.539557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:47.539523 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:55:54.139472 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.139436 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw"] Oct 30 21:55:54.140032 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.139665 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" podUID="5142aeba-c339-44d3-aa6a-4c3636f46950" containerName="maas-api" containerID="cri-o://cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea" gracePeriod=30 Oct 30 21:55:54.453992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.453965 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:55:54.482126 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.482097 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca\") pod \"5142aeba-c339-44d3-aa6a-4c3636f46950\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " Oct 30 21:55:54.482296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.482153 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs\") pod \"5142aeba-c339-44d3-aa6a-4c3636f46950\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " Oct 30 21:55:54.482296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.482272 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\") pod \"5142aeba-c339-44d3-aa6a-4c3636f46950\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " Oct 30 21:55:54.482418 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.482393 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg\" (UniqueName: \"kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg\") pod \"5142aeba-c339-44d3-aa6a-4c3636f46950\" (UID: \"5142aeba-c339-44d3-aa6a-4c3636f46950\") " Oct 30 21:55:54.482484 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.482455 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca" pod "5142aeba-c339-44d3-aa6a-4c3636f46950" (UID: "5142aeba-c339-44d3-aa6a-4c3636f46950"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:54.482859 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.482840 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5142aeba-c339-44d3-aa6a-4c3636f46950-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:54.484528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.484503 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls" pod "5142aeba-c339-44d3-aa6a-4c3636f46950" (UID: "5142aeba-c339-44d3-aa6a-4c3636f46950"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:54.484621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.484531 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs" pod "5142aeba-c339-44d3-aa6a-4c3636f46950" (UID: "5142aeba-c339-44d3-aa6a-4c3636f46950"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:54.484621 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.484549 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg" pod "5142aeba-c339-44d3-aa6a-4c3636f46950" (UID: "5142aeba-c339-44d3-aa6a-4c3636f46950"). InnerVolumeSpecName "kube-api-access-nvfsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:54.584587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.584552 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:54.584587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.584584 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5142aeba-c339-44d3-aa6a-4c3636f46950-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:54.584789 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.584595 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfsg\" (UniqueName: \"kubernetes.io/projected/5142aeba-c339-44d3-aa6a-4c3636f46950-kube-api-access-nvfsg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:54.864543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.864457 5914 generic.go:356] "Generic (PLEG): container finished" podID="5142aeba-c339-44d3-aa6a-4c3636f46950" containerID="cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea" exitCode=0 Oct 30 21:55:54.864543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.864504 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" Oct 30 21:55:54.864743 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.864539 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" event={"ID":"5142aeba-c339-44d3-aa6a-4c3636f46950","Type":"ContainerDied","Data":"cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea"} Oct 30 21:55:54.864743 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.864596 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw" event={"ID":"5142aeba-c339-44d3-aa6a-4c3636f46950","Type":"ContainerDied","Data":"607e60c81f0df3eddcf61b1bedb89f4dee9f8c4abcadedf162daf3aa5e80b2f7"} Oct 30 21:55:54.864743 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.864618 5914 scope.go:122] "RemoveContainer" containerID="cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea" Oct 30 21:55:54.875269 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.875252 5914 scope.go:122] "RemoveContainer" containerID="cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea" Oct 30 21:55:54.875569 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:55:54.875551 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea\": container with ID starting with cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea not found: ID does not exist" containerID="cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea" Oct 30 21:55:54.875612 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:54.875579 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea"} err="failed to get container status \"cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea\": rpc error: code = NotFound desc = could not find container \"cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea\": container with ID starting with cac71271099aa1b12b43316417ffbb4423067efc22fc912dd568778d1a11c8ea not found: ID does not exist" Oct 30 21:55:55.032296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:55.032262 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw"] Oct 30 21:55:55.036758 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:55.036720 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-89c0fd-5f8dfb569c-b2jgw"] Oct 30 21:55:56.217855 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:56.217819 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5142aeba-c339-44d3-aa6a-4c3636f46950" path="/var/lib/kubelet/pods/5142aeba-c339-44d3-aa6a-4c3636f46950/volumes" Oct 30 21:55:56.643953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:56.643920 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5"] Oct 30 21:55:56.644183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:56.644158 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" podUID="6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" containerName="maas-api" containerID="cri-o://ed826b12af4c82de5282cb445da822885c455e0f6af4a1fc6d95fd4fe2be87ff" gracePeriod=30 Oct 30 21:55:56.913954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:56.913851 5914 generic.go:356] "Generic (PLEG): container finished" podID="6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" containerID="ed826b12af4c82de5282cb445da822885c455e0f6af4a1fc6d95fd4fe2be87ff" exitCode=0 Oct 30 21:55:56.914119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:56.913964 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" event={"ID":"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c","Type":"ContainerDied","Data":"ed826b12af4c82de5282cb445da822885c455e0f6af4a1fc6d95fd4fe2be87ff"} Oct 30 21:55:56.989893 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:56.989844 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:55:57.087965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.061227 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs\") pod \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " Oct 30 21:55:57.087965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.079162 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca\") pod \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " Oct 30 21:55:57.087965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.079261 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\") pod \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " Oct 30 21:55:57.087965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.079295 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z\" (UniqueName: \"kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z\") pod \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\" (UID: \"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c\") " Oct 30 21:55:57.087965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.080104 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca" pod "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" (UID: "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:57.087965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.085990 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs" pod "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" (UID: "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:57.088845 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.088796 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls" pod "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" (UID: "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:57.095274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.095243 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z" pod "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" (UID: "6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c"). InnerVolumeSpecName "kube-api-access-87d7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:57.181664 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.181582 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.181664 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.181613 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.181664 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.181623 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.181664 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.181632 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87d7z\" (UniqueName: \"kubernetes.io/projected/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c-kube-api-access-87d7z\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:55:57.922868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.922793 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" Oct 30 21:55:57.922868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.922818 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5" event={"ID":"6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c","Type":"ContainerDied","Data":"9acf5f1fd4fae8cba3ebaf1e3479ed990a94c18b5af016fe1b72a5dac3f5b22e"} Oct 30 21:55:57.923371 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:57.922915 5914 scope.go:122] "RemoveContainer" containerID="ed826b12af4c82de5282cb445da822885c455e0f6af4a1fc6d95fd4fe2be87ff" Oct 30 21:55:58.117985 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:58.117937 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5"] Oct 30 21:55:58.168113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:58.168076 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3ddec2-68bbc7b9cc-hnnx5"] Oct 30 21:55:58.218107 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:55:58.218027 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" path="/var/lib/kubelet/pods/6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c/volumes" Oct 30 21:56:03.242267 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:03.242233 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd"] Oct 30 21:56:03.242702 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:03.242458 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" containerID="cri-o://6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b" gracePeriod=30 Oct 30 21:56:05.482498 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:05.482458 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr"] Oct 30 21:56:05.483056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:05.482711 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" containerID="cri-o://6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7" gracePeriod=30 Oct 30 21:56:05.496729 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:05.496695 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj"] Oct 30 21:56:05.496940 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:05.496917 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" containerID="cri-o://384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9" gracePeriod=30 Oct 30 21:56:08.368703 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.368664 5914 patch_prober.go:29] interesting pod/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:56:08.369093 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.368740 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:56:08.520898 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.520817 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:56:08.644510 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.644476 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.644676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.644549 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.644676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.644613 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.644676 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.644665 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.644865 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.644844 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.644975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.644954 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.645035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645009 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.645369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645091 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:08.645369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645152 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.645369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645200 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs\") pod \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\" (UID: \"c0bc112e-9447-45bb-a7de-13694d0b4c2d\") " Oct 30 21:56:08.645369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645323 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:08.645913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645505 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:08.645913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645517 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:08.645913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.645800 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:08.646102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.646054 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.646102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.646078 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.646102 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.646092 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.646270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.646107 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.646270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.646120 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.647003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.646976 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:08.647095 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.647020 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:08.647254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.647233 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "kube-api-access-h8xkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:08.647513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.647490 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token" pod "c0bc112e-9447-45bb-a7de-13694d0b4c2d" (UID: "c0bc112e-9447-45bb-a7de-13694d0b4c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:08.747966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.747933 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.747966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.747964 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.748161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.747977 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bc112e-9447-45bb-a7de-13694d0b4c2d-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.748161 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.747991 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8xkz\" (UniqueName: \"kubernetes.io/projected/c0bc112e-9447-45bb-a7de-13694d0b4c2d-kube-api-access-h8xkz\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:08.991167 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.991135 5914 generic.go:356] "Generic (PLEG): container finished" podID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerID="6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b" exitCode=0 Oct 30 21:56:08.991377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.991188 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" Oct 30 21:56:08.991377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.991216 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" event={"ID":"c0bc112e-9447-45bb-a7de-13694d0b4c2d","Type":"ContainerDied","Data":"6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b"} Oct 30 21:56:08.991377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.991283 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd" event={"ID":"c0bc112e-9447-45bb-a7de-13694d0b4c2d","Type":"ContainerDied","Data":"6f2e2b9cd0e5fe62776350d1d75e1c41b23237edeb8984d94a66cb61c7863c00"} Oct 30 21:56:08.991377 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:08.991315 5914 scope.go:122] "RemoveContainer" containerID="6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b" Oct 30 21:56:09.002566 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:09.002549 5914 scope.go:122] "RemoveContainer" containerID="6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b" Oct 30 21:56:09.002824 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:09.002805 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b\": container with ID starting with 6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b not found: ID does not exist" containerID="6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b" Oct 30 21:56:09.002873 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:09.002832 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b"} err="failed to get container status \"6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b\": rpc error: code = NotFound desc = could not find container \"6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b\": container with ID starting with 6dfe400d1ff02623356411d879d4015a75965db2ff495261906401e418ebf94b not found: ID does not exist" Oct 30 21:56:09.157220 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:09.157186 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd"] Oct 30 21:56:09.206416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:09.206383 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-89c0fd-openshift-default-6877577c49-j7nbd"] Oct 30 21:56:09.743528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:09.743495 5914 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:09.743936 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:09.743549 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:10.216736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.216699 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" path="/var/lib/kubelet/pods/c0bc112e-9447-45bb-a7de-13694d0b4c2d/volumes" Oct 30 21:56:10.770386 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.770361 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:56:10.798727 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.798702 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:56:10.876522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876491 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.876680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876537 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.876680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876561 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.876680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876589 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.876680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876627 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.876680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876656 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876686 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876718 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.877014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876764 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.877014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876838 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877014 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.876873 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877032 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.877270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877068 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877092 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:10.877270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877145 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.877270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877201 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877244 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.877745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877277 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.877745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877308 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket\") pod \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\" (UID: \"c4b41ca1-c2ff-4283-8f28-e7a2199b885f\") " Oct 30 21:56:10.877745 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877413 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert\") pod \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\" (UID: \"0b0f307a-eec1-49f8-86b3-8cb9220c6313\") " Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.877967 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878053 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878198 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878375 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878340 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878606 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878625 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.878790 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879047 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879066 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879080 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879101 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879115 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879129 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879141 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879154 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879168 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.879359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879180 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.880433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879501 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "kube-api-access-v5zm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.880433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879838 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.880433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.879861 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.880433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.880092 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.880433 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.880316 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:10.880721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.880451 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo" pod "c4b41ca1-c2ff-4283-8f28-e7a2199b885f" (UID: "c4b41ca1-c2ff-4283-8f28-e7a2199b885f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:10.881193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.881175 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.881288 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.881272 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8" pod "0b0f307a-eec1-49f8-86b3-8cb9220c6313" (UID: "0b0f307a-eec1-49f8-86b3-8cb9220c6313"). InnerVolumeSpecName "kube-api-access-6fmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.894963 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.894935 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k"] Oct 30 21:56:10.980766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980735 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.980766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980766 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmc8\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-kube-api-access-6fmc8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.980766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980776 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.981040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980785 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.981040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980800 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.981040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980811 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.981040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980820 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5zm8\" (UniqueName: \"kubernetes.io/projected/c4b41ca1-c2ff-4283-8f28-e7a2199b885f-kube-api-access-v5zm8\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:10.981040 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:10.980828 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0f307a-eec1-49f8-86b3-8cb9220c6313-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:11.006962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.006842 5914 generic.go:356] "Generic (PLEG): container finished" podID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerID="384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9" exitCode=0 Oct 30 21:56:11.006962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.006932 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" event={"ID":"0b0f307a-eec1-49f8-86b3-8cb9220c6313","Type":"ContainerDied","Data":"384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9"} Oct 30 21:56:11.006962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.006945 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" Oct 30 21:56:11.007262 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.007014 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj" event={"ID":"0b0f307a-eec1-49f8-86b3-8cb9220c6313","Type":"ContainerDied","Data":"d6cdaa84a47dcf6ae2c9063cb32b9251823b2bc5d470a8bf49b7b488a10252b5"} Oct 30 21:56:11.007262 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.007045 5914 scope.go:122] "RemoveContainer" containerID="384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9" Oct 30 21:56:11.009251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.009226 5914 generic.go:356] "Generic (PLEG): container finished" podID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerID="6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7" exitCode=0 Oct 30 21:56:11.009376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.009294 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" event={"ID":"c4b41ca1-c2ff-4283-8f28-e7a2199b885f","Type":"ContainerDied","Data":"6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7"} Oct 30 21:56:11.009376 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.009354 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" event={"ID":"c4b41ca1-c2ff-4283-8f28-e7a2199b885f","Type":"ContainerDied","Data":"6295ca7782cdd975cb96f637cd6651951eb28728db372f0764d8abda825a0a00"} Oct 30 21:56:11.009473 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.009383 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr" Oct 30 21:56:11.017980 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.017957 5914 scope.go:122] "RemoveContainer" containerID="384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9" Oct 30 21:56:11.018408 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:11.018304 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9\": container with ID starting with 384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9 not found: ID does not exist" containerID="384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9" Oct 30 21:56:11.018408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.018348 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9"} err="failed to get container status \"384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9\": rpc error: code = NotFound desc = could not find container \"384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9\": container with ID starting with 384c67b81cc8db913d8300cdc64b35d6ed1fe06fc2a6a6bc3962274417d501c9 not found: ID does not exist" Oct 30 21:56:11.018408 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.018372 5914 scope.go:122] "RemoveContainer" containerID="6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7" Oct 30 21:56:11.032212 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.032173 5914 scope.go:122] "RemoveContainer" containerID="6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7" Oct 30 21:56:11.034782 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:11.034749 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7\": container with ID starting with 6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7 not found: ID does not exist" containerID="6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7" Oct 30 21:56:11.034946 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.034801 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7"} err="failed to get container status \"6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7\": rpc error: code = NotFound desc = could not find container \"6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7\": container with ID starting with 6e2cac46516853575d29e28b89261cec89721a1be657a41cbd281121232f53e7 not found: ID does not exist" Oct 30 21:56:11.296799 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.296709 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr"] Oct 30 21:56:11.300416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.300387 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-csfvr"] Oct 30 21:56:11.409767 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.409731 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj"] Oct 30 21:56:11.413035 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:11.413009 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ddec2-openshift-default-6b7895b7b5-lclgj"] Oct 30 21:56:12.217450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:12.217418 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" path="/var/lib/kubelet/pods/0b0f307a-eec1-49f8-86b3-8cb9220c6313/volumes" Oct 30 21:56:12.218002 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:12.217987 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" path="/var/lib/kubelet/pods/c4b41ca1-c2ff-4283-8f28-e7a2199b885f/volumes" Oct 30 21:56:12.345436 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:12.345395 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:56:20.676218 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.676173 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx"] Oct 30 21:56:20.677016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.676981 5914 cpu_manager.go:414] "containerMap: removing container" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" Oct 30 21:56:20.677016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677002 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" Oct 30 21:56:20.677016 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677019 5914 cpu_manager.go:414] "containerMap: removing container" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677027 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677039 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" containerName="maas-api" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677047 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" containerName="maas-api" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677061 5914 cpu_manager.go:414] "containerMap: removing container" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677066 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677100 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6f7c967b-3dad-4950-b77e-005dfb518f9c" containerName="payload-pre-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677105 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7c967b-3dad-4950-b77e-005dfb518f9c" containerName="payload-pre-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677115 5914 cpu_manager.go:414] "containerMap: removing container" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677119 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677129 5914 cpu_manager.go:414] "containerMap: removing container" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677133 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677140 5914 cpu_manager.go:414] "containerMap: removing container" podUID="5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" containerName="payload-pre-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677148 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" containerName="payload-pre-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677166 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677172 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677181 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ed6110a5-3d09-4d18-a591-be31f5c5e1d4" containerName="payload-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677186 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6110a5-3d09-4d18-a591-be31f5c5e1d4" containerName="payload-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677195 5914 cpu_manager.go:414] "containerMap: removing container" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677201 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677208 5914 cpu_manager.go:414] "containerMap: removing container" podUID="982d2a1c-eacf-41cf-b9d7-649838190b31" containerName="maas-api" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677212 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982d2a1c-eacf-41cf-b9d7-649838190b31" containerName="maas-api" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677224 5914 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677231 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677241 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677245 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677256 5914 cpu_manager.go:414] "containerMap: removing container" podUID="5142aeba-c339-44d3-aa6a-4c3636f46950" containerName="maas-api" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677260 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5142aeba-c339-44d3-aa6a-4c3636f46950" containerName="maas-api" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677268 5914 cpu_manager.go:414] "containerMap: removing container" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677274 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677287 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677295 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677310 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677317 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677327 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ef2ee665-ffed-422b-a220-6304d47f1800" containerName="payload-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677332 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2ee665-ffed-422b-a220-6304d47f1800" containerName="payload-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677341 5914 cpu_manager.go:414] "containerMap: removing container" podUID="d13b974f-3697-4db8-85df-047cc6c7aab7" containerName="payload-pre-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677345 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13b974f-3697-4db8-85df-047cc6c7aab7" containerName="payload-pre-processing" Oct 30 21:56:20.677333 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677352 5914 cpu_manager.go:414] "containerMap: removing container" podUID="59bbd2ec-0a1e-406a-945a-a271d0e66bc5" containerName="maas-api" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677358 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bbd2ec-0a1e-406a-945a-a271d0e66bc5" containerName="maas-api" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677459 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f307a-eec1-49f8-86b3-8cb9220c6313" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677472 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bc112e-9447-45bb-a7de-13694d0b4c2d" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677486 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5142aeba-c339-44d3-aa6a-4c3636f46950" containerName="maas-api" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677495 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e554433-27a6-4edb-bc0e-416f50b6bb87" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677503 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39efa30-1507-4ebb-a488-1dd88d1a22a6" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677514 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6369fbb0-e0ba-43b4-92ec-c8f2afae70b9" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677530 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2ee665-ffed-422b-a220-6304d47f1800" containerName="payload-processing" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677538 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13b974f-3697-4db8-85df-047cc6c7aab7" containerName="payload-pre-processing" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677547 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bbd2ec-0a1e-406a-945a-a271d0e66bc5" containerName="maas-api" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677558 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d2491f-fbab-4843-9793-17fbd81dcb8c" containerName="main" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677564 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4cee3d-3fd2-453a-b01b-0499c6d2a773" containerName="main" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677571 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9d8b7b-d3b5-467d-a1d7-18410ef5e47c" containerName="maas-api" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677582 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677594 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7c967b-3dad-4950-b77e-005dfb518f9c" containerName="payload-pre-processing" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677601 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f3aaea-fd91-4970-8bb6-2bc08ecdf22e" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677610 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b30a216-a370-463e-b9c9-125427eec2fb" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677619 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c150d9e-7846-4ebd-82d0-2ed3790a4ecb" containerName="payload-pre-processing" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677628 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677636 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b41ca1-c2ff-4283-8f28-e7a2199b885f" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677645 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6110a5-3d09-4d18-a591-be31f5c5e1d4" containerName="payload-processing" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677655 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fad0ca5-639a-474f-aa48-51184c8adbf8" containerName="istio-proxy" Oct 30 21:56:20.679116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.677661 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="982d2a1c-eacf-41cf-b9d7-649838190b31" containerName="maas-api" Oct 30 21:56:20.681475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.681456 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.705264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.705230 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jd9\" (UniqueName: \"kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.705264 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.705267 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.705489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.705286 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.705489 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.705401 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.735903 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.735852 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx"] Oct 30 21:56:20.782130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.782096 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-75c2acf6-serving-cert\"" Oct 30 21:56:20.806263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.806227 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jd9\" (UniqueName: \"kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.806263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.806269 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.806465 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.806285 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.806465 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.806311 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.806465 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:20.806443 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-serving-cert: secret "maas-api-e2e-mt-75c2acf6-serving-cert" not found Oct 30 21:56:20.806587 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:20.806503 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls podName:447cbf22-f34e-4c95-ac87-127d1f9ad06c nodeName:}" failed. No retries permitted until 2026-10-30 21:56:21.306487612 +0000 UTC m=+2333.208351656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls") pod "maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" (UID: "447cbf22-f34e-4c95-ac87-127d1f9ad06c") : secret "maas-api-e2e-mt-75c2acf6-serving-cert" not found Oct 30 21:56:20.807013 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.806982 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.808919 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.808899 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:20.863616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:20.863576 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jd9\" (UniqueName: \"kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:21.313128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:21.313089 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:21.315714 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:21.315685 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:21.597560 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:21.597452 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:21.974210 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:21.974182 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx"] Oct 30 21:56:21.975858 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:56:21.975824 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447cbf22_f34e_4c95_ac87_127d1f9ad06c.slice/crio-efb4d1143a30666fe6578c9625e09a668512f26c6f822121fa3a979c318045b1 WatchSource:0}: Error finding container efb4d1143a30666fe6578c9625e09a668512f26c6f822121fa3a979c318045b1: Status 404 returned error can't find the container with id efb4d1143a30666fe6578c9625e09a668512f26c6f822121fa3a979c318045b1 Oct 30 21:56:22.086193 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:22.086151 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" event={"ID":"447cbf22-f34e-4c95-ac87-127d1f9ad06c","Type":"ContainerStarted","Data":"efb4d1143a30666fe6578c9625e09a668512f26c6f822121fa3a979c318045b1"} Oct 30 21:56:23.094420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:23.094386 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" event={"ID":"447cbf22-f34e-4c95-ac87-127d1f9ad06c","Type":"ContainerStarted","Data":"ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c"} Oct 30 21:56:23.094811 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:23.094475 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:23.211425 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:23.211372 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" podStartSLOduration=2.327323559 podStartE2EDuration="3.211353987s" totalImagesPullingTime="884.030428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:23.210140105 +0000 UTC m=+2335.112004172" watchObservedRunningTime="2026-10-30 21:56:23.211353987 +0000 UTC m=+2335.113218050" Oct 30 21:56:25.916054 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:25.915988 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" containerID="cri-o://15e8bfe71f867c7d6f0511c7b6d5a6602610a4545afd7ba203a13ef11e71c898" gracePeriod=45 Oct 30 21:56:26.116736 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.116687 5914 generic.go:356] "Generic (PLEG): container finished" podID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerID="15e8bfe71f867c7d6f0511c7b6d5a6602610a4545afd7ba203a13ef11e71c898" exitCode=0 Oct 30 21:56:26.116932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.116759 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" event={"ID":"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d","Type":"ContainerDied","Data":"15e8bfe71f867c7d6f0511c7b6d5a6602610a4545afd7ba203a13ef11e71c898"} Oct 30 21:56:26.181546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.181473 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:56:26.316448 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.316394 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm\") pod \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " Oct 30 21:56:26.317196 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.317162 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir\") pod \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " Oct 30 21:56:26.317444 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.317425 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs\") pod \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " Oct 30 21:56:26.317690 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.317671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home\") pod \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " Oct 30 21:56:26.317876 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.317857 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache\") pod \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " Oct 30 21:56:26.323998 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.323971 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls\" (UniqueName: \"kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls\") pod \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\" (UID: \"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d\") " Oct 30 21:56:26.333807 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.320902 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm" pod "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" (UID: "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.333807 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.323237 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir" pod "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" (UID: "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.333807 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.327954 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home" pod "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" (UID: "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.333807 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.329041 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache" pod "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" (UID: "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:26.340830 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.340791 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs" pod "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" (UID: "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:26.341168 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.341143 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls" pod "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" (UID: "c8868cef-5ba6-485c-b2b5-f66b52d6bd0d"). InnerVolumeSpecName "kube-api-access-2dcls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:26.425853 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.425811 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.425853 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.425850 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.425853 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.425859 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.426139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.425868 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.426139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.425894 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dcls\" (UniqueName: \"kubernetes.io/projected/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-kube-api-access-2dcls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:26.426139 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:26.425903 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:27.124237 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:27.124206 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" Oct 30 21:56:27.124725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:27.124201 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k" event={"ID":"c8868cef-5ba6-485c-b2b5-f66b52d6bd0d","Type":"ContainerDied","Data":"e3f302f80936389fa984c745f40008aad6e7c37dbb6e550de3102e9d23182e5b"} Oct 30 21:56:27.124725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:27.124329 5914 scope.go:122] "RemoveContainer" containerID="15e8bfe71f867c7d6f0511c7b6d5a6602610a4545afd7ba203a13ef11e71c898" Oct 30 21:56:27.287493 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:27.287455 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k"] Oct 30 21:56:27.336904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:27.336847 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e4dadd/distinct-w1-e4dadd-kserve-669f669478-dhw5k"] Oct 30 21:56:28.281041 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:28.281008 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" path="/var/lib/kubelet/pods/c8868cef-5ba6-485c-b2b5-f66b52d6bd0d/volumes" Oct 30 21:56:29.106860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:29.106833 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 21:56:31.579720 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:31.579683 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz"] Oct 30 21:56:31.580186 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:31.579911 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" containerID="cri-o://17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef" gracePeriod=30 Oct 30 21:56:31.722956 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:31.722924 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:31.723121 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:31.722998 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:36.836319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.836293 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:56:36.877028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877003 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877057 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877114 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877177 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877138 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877185 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877328 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877249 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877328 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:36.877423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877369 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877538 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877421 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:36.877592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877568 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877606 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs\") pod \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\" (UID: \"83b53dc9-b989-4d3d-9c2f-15ce03381d63\") " Oct 30 21:56:36.877655 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877544 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:36.877766 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877708 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:36.877951 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.877927 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:36.878201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.878152 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.878201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.878182 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.878201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.878196 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.878438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.878209 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.878438 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.878222 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.879685 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.879651 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:36.879784 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.879746 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:36.880218 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.880193 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "kube-api-access-xpgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:36.880344 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.880327 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo" pod "83b53dc9-b989-4d3d-9c2f-15ce03381d63" (UID: "83b53dc9-b989-4d3d-9c2f-15ce03381d63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:36.979319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.979289 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.979319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.979316 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.979532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.979327 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpgzm\" (UniqueName: \"kubernetes.io/projected/83b53dc9-b989-4d3d-9c2f-15ce03381d63-kube-api-access-xpgzm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:36.979532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:36.979340 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b53dc9-b989-4d3d-9c2f-15ce03381d63-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:37.186296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.186205 5914 generic.go:356] "Generic (PLEG): container finished" podID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerID="17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef" exitCode=0 Oct 30 21:56:37.186296 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.186255 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" Oct 30 21:56:37.186532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.186298 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" event={"ID":"83b53dc9-b989-4d3d-9c2f-15ce03381d63","Type":"ContainerDied","Data":"17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef"} Oct 30 21:56:37.186532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.186387 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz" event={"ID":"83b53dc9-b989-4d3d-9c2f-15ce03381d63","Type":"ContainerDied","Data":"10e1e9f817aad48e51501873d90da6c1d6a5213ae2c961589e3be5f177b1be0f"} Oct 30 21:56:37.186532 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.186417 5914 scope.go:122] "RemoveContainer" containerID="17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef" Oct 30 21:56:37.196831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.196815 5914 scope.go:122] "RemoveContainer" containerID="17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef" Oct 30 21:56:37.197119 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:37.197102 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef\": container with ID starting with 17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef not found: ID does not exist" containerID="17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef" Oct 30 21:56:37.197156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.197128 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef"} err="failed to get container status \"17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef\": rpc error: code = NotFound desc = could not find container \"17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef\": container with ID starting with 17a29e175715e6268394e88962da2b15307231e73e14da8e83e0b9f275aa9fef not found: ID does not exist" Oct 30 21:56:37.349449 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.349417 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz"] Oct 30 21:56:37.356522 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:37.356493 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-k5wlz"] Oct 30 21:56:38.241504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.241473 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" path="/var/lib/kubelet/pods/83b53dc9-b989-4d3d-9c2f-15ce03381d63/volumes" Oct 30 21:56:38.296391 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.296354 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl"] Oct 30 21:56:38.297243 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297219 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" Oct 30 21:56:38.297323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297247 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" Oct 30 21:56:38.297323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297281 5914 cpu_manager.go:414] "containerMap: removing container" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" Oct 30 21:56:38.297323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297289 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" Oct 30 21:56:38.297425 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297324 5914 cpu_manager.go:414] "containerMap: removing container" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" Oct 30 21:56:38.297425 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297332 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b8a9d0-0196-4020-b114-73f3ffa1e7a9" containerName="istio-proxy" Oct 30 21:56:38.297500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297482 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b53dc9-b989-4d3d-9c2f-15ce03381d63" containerName="istio-proxy" Oct 30 21:56:38.297541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.297505 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8868cef-5ba6-485c-b2b5-f66b52d6bd0d" containerName="main" Oct 30 21:56:38.302667 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.302641 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.349894 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.349863 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl"] Oct 30 21:56:38.354721 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.354691 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dbe6da\"" Oct 30 21:56:38.396366 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.396341 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dbe6da-dockercfg-sd4zf\"" Oct 30 21:56:38.404546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.404521 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.404739 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.404721 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqsx\" (UniqueName: \"kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx\") pod \"payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.505636 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.505549 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqsx\" (UniqueName: \"kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx\") pod \"payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.505807 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.505648 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.506358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.506337 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.562757 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.562719 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqsx\" (UniqueName: \"kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx\") pod \"payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.617953 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.617898 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:38.997469 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:38.997441 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl"] Oct 30 21:56:38.999180 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:56:38.999150 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a01a558_919c_449b_861b_aed401f09b25.slice/crio-752a425809978e34f0b9c803208c0086e6da5b310e50147e686008205360ad4c WatchSource:0}: Error finding container 752a425809978e34f0b9c803208c0086e6da5b310e50147e686008205360ad4c: Status 404 returned error can't find the container with id 752a425809978e34f0b9c803208c0086e6da5b310e50147e686008205360ad4c Oct 30 21:56:39.200954 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:39.200917 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" event={"ID":"4a01a558-919c-449b-861b-aed401f09b25","Type":"ContainerStarted","Data":"ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd"} Oct 30 21:56:39.201135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:39.200975 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" event={"ID":"4a01a558-919c-449b-861b-aed401f09b25","Type":"ContainerStarted","Data":"752a425809978e34f0b9c803208c0086e6da5b310e50147e686008205360ad4c"} Oct 30 21:56:39.201135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:39.201127 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:39.320025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:39.319900 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" podStartSLOduration=1.319870135 podStartE2EDuration="1.319870135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:39.318091741 +0000 UTC m=+2351.219955807" watchObservedRunningTime="2026-10-30 21:56:39.319870135 +0000 UTC m=+2351.221734199" Oct 30 21:56:40.664070 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:40.664037 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:56:42.346570 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:42.346531 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:56:43.273592 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:43.273556 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv"] Oct 30 21:56:46.765119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:46.765085 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g"] Oct 30 21:56:46.870255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:46.870219 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5"] Oct 30 21:56:50.216724 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:50.216689 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 21:56:50.487624 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:50.487545 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd"] Oct 30 21:56:50.599041 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:50.598979 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94"] Oct 30 21:56:51.533248 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:51.533210 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp"] Oct 30 21:56:51.534156 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:51.534102 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" podUID="78c32271-7a11-4546-92bf-2706c934fa12" containerName="maas-api" containerID="cri-o://ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0" gracePeriod=30 Oct 30 21:56:51.771251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:51.771005 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" podUID="ef71a298-f0f5-4284-8059-245296585782" containerName="payload-processing" containerID="cri-o://bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c" gracePeriod=25 Oct 30 21:56:51.871025 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:51.870991 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" podUID="0edca360-fbf2-4f5e-93dd-54e7c7910347" containerName="payload-pre-processing" containerID="cri-o://7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe" gracePeriod=25 Oct 30 21:56:51.984759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:51.984728 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:56:52.021089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.020072 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\") pod \"78c32271-7a11-4546-92bf-2706c934fa12\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " Oct 30 21:56:52.021089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.020120 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh\" (UniqueName: \"kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh\") pod \"78c32271-7a11-4546-92bf-2706c934fa12\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " Oct 30 21:56:52.021089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.020158 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs\") pod \"78c32271-7a11-4546-92bf-2706c934fa12\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " Oct 30 21:56:52.021089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.020192 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca\") pod \"78c32271-7a11-4546-92bf-2706c934fa12\" (UID: \"78c32271-7a11-4546-92bf-2706c934fa12\") " Oct 30 21:56:52.021089 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.020838 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca" pod "78c32271-7a11-4546-92bf-2706c934fa12" (UID: "78c32271-7a11-4546-92bf-2706c934fa12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:52.024332 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.024292 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls" pod "78c32271-7a11-4546-92bf-2706c934fa12" (UID: "78c32271-7a11-4546-92bf-2706c934fa12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:52.025118 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.025086 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs" pod "78c32271-7a11-4546-92bf-2706c934fa12" (UID: "78c32271-7a11-4546-92bf-2706c934fa12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:52.027671 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.027471 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh" pod "78c32271-7a11-4546-92bf-2706c934fa12" (UID: "78c32271-7a11-4546-92bf-2706c934fa12"). InnerVolumeSpecName "kube-api-access-8vjrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:52.127485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.126996 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.127485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.127035 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjrh\" (UniqueName: \"kubernetes.io/projected/78c32271-7a11-4546-92bf-2706c934fa12-kube-api-access-8vjrh\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.127485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.127049 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78c32271-7a11-4546-92bf-2706c934fa12-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.127485 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.127062 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c32271-7a11-4546-92bf-2706c934fa12-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.178593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.178564 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:56:52.210738 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.210712 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:56:52.227585 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.227559 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume\") pod \"ef71a298-f0f5-4284-8059-245296585782\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " Oct 30 21:56:52.227709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.227666 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5\" (UniqueName: \"kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5\") pod \"ef71a298-f0f5-4284-8059-245296585782\" (UID: \"ef71a298-f0f5-4284-8059-245296585782\") " Oct 30 21:56:52.228004 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.227979 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume" pod "ef71a298-f0f5-4284-8059-245296585782" (UID: "ef71a298-f0f5-4284-8059-245296585782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:52.228119 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.228101 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef71a298-f0f5-4284-8059-245296585782-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.229986 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.229965 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5" pod "ef71a298-f0f5-4284-8059-245296585782" (UID: "ef71a298-f0f5-4284-8059-245296585782"). InnerVolumeSpecName "kube-api-access-zvgn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:52.325339 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.325230 5914 generic.go:356] "Generic (PLEG): container finished" podID="0edca360-fbf2-4f5e-93dd-54e7c7910347" containerID="7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe" exitCode=0 Oct 30 21:56:52.325546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.325376 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" event={"ID":"0edca360-fbf2-4f5e-93dd-54e7c7910347","Type":"ContainerDied","Data":"7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe"} Oct 30 21:56:52.325546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.325431 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" event={"ID":"0edca360-fbf2-4f5e-93dd-54e7c7910347","Type":"ContainerDied","Data":"c6e48ae81c46caed0f55c32a0707e6f79969868f870041602ea5f257ebacfb8d"} Oct 30 21:56:52.325546 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.325459 5914 scope.go:122] "RemoveContainer" containerID="7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe" Oct 30 21:56:52.325835 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.325631 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5" Oct 30 21:56:52.328817 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.328787 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx\" (UniqueName: \"kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx\") pod \"0edca360-fbf2-4f5e-93dd-54e7c7910347\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " Oct 30 21:56:52.328997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.328845 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume\") pod \"0edca360-fbf2-4f5e-93dd-54e7c7910347\" (UID: \"0edca360-fbf2-4f5e-93dd-54e7c7910347\") " Oct 30 21:56:52.328997 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.328970 5914 generic.go:356] "Generic (PLEG): container finished" podID="ef71a298-f0f5-4284-8059-245296585782" containerID="bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c" exitCode=0 Oct 30 21:56:52.329123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.329075 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" event={"ID":"ef71a298-f0f5-4284-8059-245296585782","Type":"ContainerDied","Data":"bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c"} Oct 30 21:56:52.329123 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.329116 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" event={"ID":"ef71a298-f0f5-4284-8059-245296585782","Type":"ContainerDied","Data":"0a7e13a16436df81e7c53c7e077bcd135371c66f6f6223caa8e70b6e6b52496d"} Oct 30 21:56:52.329235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.329178 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g" Oct 30 21:56:52.329598 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.329573 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvgn5\" (UniqueName: \"kubernetes.io/projected/ef71a298-f0f5-4284-8059-245296585782-kube-api-access-zvgn5\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.329790 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.329764 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume" pod "0edca360-fbf2-4f5e-93dd-54e7c7910347" (UID: "0edca360-fbf2-4f5e-93dd-54e7c7910347"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:52.331660 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.331333 5914 generic.go:356] "Generic (PLEG): container finished" podID="78c32271-7a11-4546-92bf-2706c934fa12" containerID="ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0" exitCode=0 Oct 30 21:56:52.331660 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.331422 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" event={"ID":"78c32271-7a11-4546-92bf-2706c934fa12","Type":"ContainerDied","Data":"ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0"} Oct 30 21:56:52.331660 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.331462 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" event={"ID":"78c32271-7a11-4546-92bf-2706c934fa12","Type":"ContainerDied","Data":"e052cf4754f042dbc6490188b971240faafc59460ce0e750e1ed3ac11ad3abbb"} Oct 30 21:56:52.331660 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.331534 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp" Oct 30 21:56:52.335643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.335609 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx" pod "0edca360-fbf2-4f5e-93dd-54e7c7910347" (UID: "0edca360-fbf2-4f5e-93dd-54e7c7910347"). InnerVolumeSpecName "kube-api-access-7t4hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:52.349502 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.349483 5914 scope.go:122] "RemoveContainer" containerID="7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe" Oct 30 21:56:52.349821 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:52.349800 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe\": container with ID starting with 7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe not found: ID does not exist" containerID="7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe" Oct 30 21:56:52.349895 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.349831 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe"} err="failed to get container status \"7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe\": rpc error: code = NotFound desc = could not find container \"7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe\": container with ID starting with 7fa13d79927011feeb1cd577f2abcb3ee07eb69248d279b63632b3962ab840fe not found: ID does not exist" Oct 30 21:56:52.349895 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.349848 5914 scope.go:122] "RemoveContainer" containerID="bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c" Oct 30 21:56:52.359130 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.359107 5914 scope.go:122] "RemoveContainer" containerID="bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c" Oct 30 21:56:52.359438 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:52.359416 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c\": container with ID starting with bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c not found: ID does not exist" containerID="bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c" Oct 30 21:56:52.359497 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.359446 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c"} err="failed to get container status \"bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c\": rpc error: code = NotFound desc = could not find container \"bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c\": container with ID starting with bae75b5eaf5ac5d2909ccef02f2da3434831c99a0312e9c51515728bde265e0c not found: ID does not exist" Oct 30 21:56:52.359497 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.359463 5914 scope.go:122] "RemoveContainer" containerID="ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0" Oct 30 21:56:52.368647 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.368628 5914 scope.go:122] "RemoveContainer" containerID="ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0" Oct 30 21:56:52.369030 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:52.369009 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0\": container with ID starting with ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0 not found: ID does not exist" containerID="ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0" Oct 30 21:56:52.369109 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.369039 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0"} err="failed to get container status \"ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0\": rpc error: code = NotFound desc = could not find container \"ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0\": container with ID starting with ecaf244b3ff05fe5b689b4f191ffd9b60a4b4f3106aa26585396640e2752cfd0 not found: ID does not exist" Oct 30 21:56:52.430697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.430662 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4hx\" (UniqueName: \"kubernetes.io/projected/0edca360-fbf2-4f5e-93dd-54e7c7910347-kube-api-access-7t4hx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.430697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.430698 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0edca360-fbf2-4f5e-93dd-54e7c7910347-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:52.539625 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.539596 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp"] Oct 30 21:56:52.541706 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.541685 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9796f1-5d8588d5b-bqwxp"] Oct 30 21:56:52.701137 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.701102 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g"] Oct 30 21:56:52.705144 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.705116 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b42c9-856887dbd5-mrd6g"] Oct 30 21:56:52.812235 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.812200 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5"] Oct 30 21:56:52.814126 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:52.814105 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b42c9-5b9f87fdd7-jg7x5"] Oct 30 21:56:54.218353 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:54.218320 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0edca360-fbf2-4f5e-93dd-54e7c7910347" path="/var/lib/kubelet/pods/0edca360-fbf2-4f5e-93dd-54e7c7910347/volumes" Oct 30 21:56:54.218836 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:54.218821 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c32271-7a11-4546-92bf-2706c934fa12" path="/var/lib/kubelet/pods/78c32271-7a11-4546-92bf-2706c934fa12/volumes" Oct 30 21:56:54.219247 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:54.219234 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef71a298-f0f5-4284-8059-245296585782" path="/var/lib/kubelet/pods/ef71a298-f0f5-4284-8059-245296585782/volumes" Oct 30 21:56:55.488340 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:55.488279 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" containerName="payload-processing" containerID="cri-o://d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a" gracePeriod=25 Oct 30 21:56:55.599512 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:55.599471 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerName="payload-pre-processing" containerID="cri-o://3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860" gracePeriod=25 Oct 30 21:56:58.295708 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.295576 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" containerID="cri-o://743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3" gracePeriod=45 Oct 30 21:56:58.598143 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.598116 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:56:58.703913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.703866 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir\") pod \"360d0a39-dfaf-4624-bc7a-b57d598e216b\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " Oct 30 21:56:58.703913 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.703918 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache\") pod \"360d0a39-dfaf-4624-bc7a-b57d598e216b\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " Oct 30 21:56:58.704138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.703946 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc\" (UniqueName: \"kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc\") pod \"360d0a39-dfaf-4624-bc7a-b57d598e216b\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " Oct 30 21:56:58.704138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704091 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home\") pod \"360d0a39-dfaf-4624-bc7a-b57d598e216b\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " Oct 30 21:56:58.704250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704155 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir" pod "360d0a39-dfaf-4624-bc7a-b57d598e216b" (UID: "360d0a39-dfaf-4624-bc7a-b57d598e216b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.704250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704175 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache" pod "360d0a39-dfaf-4624-bc7a-b57d598e216b" (UID: "360d0a39-dfaf-4624-bc7a-b57d598e216b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.704250 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704201 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm\") pod \"360d0a39-dfaf-4624-bc7a-b57d598e216b\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " Oct 30 21:56:58.704420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704277 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs\" (UniqueName: \"kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs\") pod \"360d0a39-dfaf-4624-bc7a-b57d598e216b\" (UID: \"360d0a39-dfaf-4624-bc7a-b57d598e216b\") " Oct 30 21:56:58.704420 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704373 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home" pod "360d0a39-dfaf-4624-bc7a-b57d598e216b" (UID: "360d0a39-dfaf-4624-bc7a-b57d598e216b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.704754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704723 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.704754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704748 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.705128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.704761 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.706411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.706383 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc" pod "360d0a39-dfaf-4624-bc7a-b57d598e216b" (UID: "360d0a39-dfaf-4624-bc7a-b57d598e216b"). InnerVolumeSpecName "kube-api-access-8hjbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:58.706535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.706443 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs" pod "360d0a39-dfaf-4624-bc7a-b57d598e216b" (UID: "360d0a39-dfaf-4624-bc7a-b57d598e216b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:58.706535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.706489 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm" pod "360d0a39-dfaf-4624-bc7a-b57d598e216b" (UID: "360d0a39-dfaf-4624-bc7a-b57d598e216b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:58.805774 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.805744 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/360d0a39-dfaf-4624-bc7a-b57d598e216b-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.805774 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.805771 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/360d0a39-dfaf-4624-bc7a-b57d598e216b-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:58.806001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:58.805782 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjbc\" (UniqueName: \"kubernetes.io/projected/360d0a39-dfaf-4624-bc7a-b57d598e216b-kube-api-access-8hjbc\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:56:59.381475 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.381442 5914 generic.go:356] "Generic (PLEG): container finished" podID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerID="743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3" exitCode=0 Oct 30 21:56:59.381966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.381487 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" Oct 30 21:56:59.381966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.381491 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" event={"ID":"360d0a39-dfaf-4624-bc7a-b57d598e216b","Type":"ContainerDied","Data":"743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3"} Oct 30 21:56:59.381966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.381583 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv" event={"ID":"360d0a39-dfaf-4624-bc7a-b57d598e216b","Type":"ContainerDied","Data":"d5bb7ccfe7a637a625f27360c4e731390bfbb0d8be4ce6634c979752d6e60795"} Oct 30 21:56:59.381966 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.381599 5914 scope.go:122] "RemoveContainer" containerID="743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3" Oct 30 21:56:59.391923 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.391899 5914 scope.go:122] "RemoveContainer" containerID="743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3" Oct 30 21:56:59.392186 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:56:59.392167 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3\": container with ID starting with 743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3 not found: ID does not exist" containerID="743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3" Oct 30 21:56:59.392263 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.392192 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3"} err="failed to get container status \"743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3\": rpc error: code = NotFound desc = could not find container \"743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3\": container with ID starting with 743f0cf93abce90c6cd427fe4fd3c87c62de93d1533b625ffe6d9c16ddf1c1c3 not found: ID does not exist" Oct 30 21:56:59.544593 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.544554 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv"] Oct 30 21:56:59.596847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:56:59.596811 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-021907/simulator-w5-021907-kserve-76d8b986c5-2dpkv"] Oct 30 21:57:00.216368 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:00.216335 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" path="/var/lib/kubelet/pods/360d0a39-dfaf-4624-bc7a-b57d598e216b/volumes" Oct 30 21:57:00.347323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:00.347288 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:57:00.941254 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:00.941215 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp"] Oct 30 21:57:00.941709 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:00.941425 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" containerID="cri-o://04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894" gracePeriod=30 Oct 30 21:57:01.353421 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:01.353343 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:57:05.531944 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:05.531904 5914 patch_prober.go:29] interesting pod/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:05.532339 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:05.531965 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:06.210087 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.210019 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:57:06.305067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305007 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305114 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305266 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305171 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305279 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305406 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305296 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:06.305535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305405 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305447 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:06.305535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305466 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305535 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305510 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305560 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305599 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy\") pod \"60bad27b-019f-4b10-b8d7-1a68b898992f\" (UID: \"60bad27b-019f-4b10-b8d7-1a68b898992f\") " Oct 30 21:57:06.305737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305561 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:06.305737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305668 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:06.305975 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.305757 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:06.306327 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.306299 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.306501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.306475 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.306614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.306503 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60bad27b-019f-4b10-b8d7-1a68b898992f-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.306614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.306518 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.306614 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.306533 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.307653 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.307621 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:06.307747 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.307727 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:06.307868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.307846 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:06.308001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.307985 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29" pod "60bad27b-019f-4b10-b8d7-1a68b898992f" (UID: "60bad27b-019f-4b10-b8d7-1a68b898992f"). InnerVolumeSpecName "kube-api-access-9td29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:06.407991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.407942 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.407991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.407978 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9td29\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-kube-api-access-9td29\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.407991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.407995 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.408307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.408007 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60bad27b-019f-4b10-b8d7-1a68b898992f-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:06.427949 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.427912 5914 generic.go:356] "Generic (PLEG): container finished" podID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerID="04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894" exitCode=0 Oct 30 21:57:06.428116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.427980 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" Oct 30 21:57:06.428116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.427987 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" event={"ID":"60bad27b-019f-4b10-b8d7-1a68b898992f","Type":"ContainerDied","Data":"04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894"} Oct 30 21:57:06.428116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.428064 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp" event={"ID":"60bad27b-019f-4b10-b8d7-1a68b898992f","Type":"ContainerDied","Data":"1a66552be5b007b34993276b611783fcdf6b44eb4b50a973ee39aad356f045f6"} Oct 30 21:57:06.428116 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.428099 5914 scope.go:122] "RemoveContainer" containerID="04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894" Oct 30 21:57:06.438794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.438597 5914 scope.go:122] "RemoveContainer" containerID="99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7" Oct 30 21:57:06.447717 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.447700 5914 scope.go:122] "RemoveContainer" containerID="04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894" Oct 30 21:57:06.448073 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:06.448050 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894\": container with ID starting with 04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894 not found: ID does not exist" containerID="04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894" Oct 30 21:57:06.448128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.448084 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894"} err="failed to get container status \"04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894\": rpc error: code = NotFound desc = could not find container \"04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894\": container with ID starting with 04210aacfb34e2703dd16cec83eca5e0f3e4f1fc22ebda56c06472fc2f52d894 not found: ID does not exist" Oct 30 21:57:06.448128 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.448103 5914 scope.go:122] "RemoveContainer" containerID="99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7" Oct 30 21:57:06.448341 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:06.448323 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7\": container with ID starting with 99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7 not found: ID does not exist" containerID="99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7" Oct 30 21:57:06.448381 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.448347 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7"} err="failed to get container status \"99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7\": rpc error: code = NotFound desc = could not find container \"99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7\": container with ID starting with 99a0c0f1a42a6007e10438f4260e727c143159e0f77b11c482d79ebd4ab577a7 not found: ID does not exist" Oct 30 21:57:06.594902 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.594803 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp"] Oct 30 21:57:06.597992 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:06.597965 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9796f1-openshift-default-5547cf6495-fsqnp"] Oct 30 21:57:08.216862 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:08.216828 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" path="/var/lib/kubelet/pods/60bad27b-019f-4b10-b8d7-1a68b898992f/volumes" Oct 30 21:57:10.346716 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:10.346676 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:57:11.354503 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:11.354465 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:57:12.345537 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:12.345484 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:57:15.746459 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.746386 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:57:15.828821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.828793 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume\") pod \"fb8eb883-8773-4f50-8f67-b3621be81239\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " Oct 30 21:57:15.829033 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.828908 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls\" (UniqueName: \"kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls\") pod \"fb8eb883-8773-4f50-8f67-b3621be81239\" (UID: \"fb8eb883-8773-4f50-8f67-b3621be81239\") " Oct 30 21:57:15.829265 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.829238 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume" pod "fb8eb883-8773-4f50-8f67-b3621be81239" (UID: "fb8eb883-8773-4f50-8f67-b3621be81239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:15.831504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.831477 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls" pod "fb8eb883-8773-4f50-8f67-b3621be81239" (UID: "fb8eb883-8773-4f50-8f67-b3621be81239"). InnerVolumeSpecName "kube-api-access-6z5ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:15.851094 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.851071 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:57:15.930176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.930146 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59\" (UniqueName: \"kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59\") pod \"ac64985c-d072-4a7b-b1a9-18733974a80b\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " Oct 30 21:57:15.930356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.930305 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume\") pod \"ac64985c-d072-4a7b-b1a9-18733974a80b\" (UID: \"ac64985c-d072-4a7b-b1a9-18733974a80b\") " Oct 30 21:57:15.930559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.930532 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8eb883-8773-4f50-8f67-b3621be81239-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:15.930559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.930548 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5ls\" (UniqueName: \"kubernetes.io/projected/fb8eb883-8773-4f50-8f67-b3621be81239-kube-api-access-6z5ls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:15.930725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.930651 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume" pod "ac64985c-d072-4a7b-b1a9-18733974a80b" (UID: "ac64985c-d072-4a7b-b1a9-18733974a80b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:15.932378 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:15.932359 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59" pod "ac64985c-d072-4a7b-b1a9-18733974a80b" (UID: "ac64985c-d072-4a7b-b1a9-18733974a80b"). InnerVolumeSpecName "kube-api-access-brt59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:16.031675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.031596 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac64985c-d072-4a7b-b1a9-18733974a80b-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:16.031675 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.031636 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brt59\" (UniqueName: \"kubernetes.io/projected/ac64985c-d072-4a7b-b1a9-18733974a80b-kube-api-access-brt59\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:16.500672 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.500619 5914 generic.go:356] "Generic (PLEG): container finished" podID="fb8eb883-8773-4f50-8f67-b3621be81239" containerID="d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a" exitCode=0 Oct 30 21:57:16.500870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.500688 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" Oct 30 21:57:16.500870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.500694 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" event={"ID":"fb8eb883-8773-4f50-8f67-b3621be81239","Type":"ContainerDied","Data":"d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a"} Oct 30 21:57:16.500870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.500823 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd" event={"ID":"fb8eb883-8773-4f50-8f67-b3621be81239","Type":"ContainerDied","Data":"09c842cea57aa9458d4395771ca82c923f660447ce8dcc5e6020bcf01ec6f067"} Oct 30 21:57:16.500870 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.500858 5914 scope.go:122] "RemoveContainer" containerID="d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a" Oct 30 21:57:16.503132 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.503102 5914 generic.go:356] "Generic (PLEG): container finished" podID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerID="3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860" exitCode=0 Oct 30 21:57:16.503268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.503144 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" event={"ID":"ac64985c-d072-4a7b-b1a9-18733974a80b","Type":"ContainerDied","Data":"3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860"} Oct 30 21:57:16.503268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.503166 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" Oct 30 21:57:16.503268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.503203 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94" event={"ID":"ac64985c-d072-4a7b-b1a9-18733974a80b","Type":"ContainerDied","Data":"7511b4ee49e2ebee1c11c503dcfd2128e3ff065c9e2fb4e682503dbe74abed53"} Oct 30 21:57:16.519414 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.519392 5914 scope.go:122] "RemoveContainer" containerID="d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a" Oct 30 21:57:16.519722 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:16.519691 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a\": container with ID starting with d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a not found: ID does not exist" containerID="d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a" Oct 30 21:57:16.519831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.519730 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a"} err="failed to get container status \"d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a\": rpc error: code = NotFound desc = could not find container \"d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a\": container with ID starting with d8e5c302d4253d1c1f830e0cebb1c506ac846fba89321d30065299649baf9d4a not found: ID does not exist" Oct 30 21:57:16.519831 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.519748 5914 scope.go:122] "RemoveContainer" containerID="3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860" Oct 30 21:57:16.528861 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.528840 5914 scope.go:122] "RemoveContainer" containerID="3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860" Oct 30 21:57:16.529195 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:16.529166 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860\": container with ID starting with 3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860 not found: ID does not exist" containerID="3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860" Oct 30 21:57:16.529252 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.529204 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860"} err="failed to get container status \"3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860\": rpc error: code = NotFound desc = could not find container \"3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860\": container with ID starting with 3fbfb823678476b25206a145010ef9e48dd5ba08539848342b649752bdb6c860 not found: ID does not exist" Oct 30 21:57:16.663844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.663809 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd"] Oct 30 21:57:16.667492 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.667462 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-021907-6fdc9848f9-lklrd"] Oct 30 21:57:16.776152 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.776075 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94"] Oct 30 21:57:16.780215 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:16.780180 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-021907-5f85877d59-wfs94"] Oct 30 21:57:18.217183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:18.217153 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" path="/var/lib/kubelet/pods/ac64985c-d072-4a7b-b1a9-18733974a80b/volumes" Oct 30 21:57:18.217583 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:18.217555 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" path="/var/lib/kubelet/pods/fb8eb883-8773-4f50-8f67-b3621be81239/volumes" Oct 30 21:57:20.804586 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.804546 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j"] Oct 30 21:57:20.805253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805233 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerName="payload-pre-processing" Oct 30 21:57:20.805253 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805254 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerName="payload-pre-processing" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805266 5914 cpu_manager.go:414] "containerMap: removing container" podUID="0edca360-fbf2-4f5e-93dd-54e7c7910347" containerName="payload-pre-processing" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805271 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0edca360-fbf2-4f5e-93dd-54e7c7910347" containerName="payload-pre-processing" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805288 5914 cpu_manager.go:414] "containerMap: removing container" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805296 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805319 5914 cpu_manager.go:414] "containerMap: removing container" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805325 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805333 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ef71a298-f0f5-4284-8059-245296585782" containerName="payload-processing" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805337 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef71a298-f0f5-4284-8059-245296585782" containerName="payload-processing" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805352 5914 cpu_manager.go:414] "containerMap: removing container" podUID="78c32271-7a11-4546-92bf-2706c934fa12" containerName="maas-api" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805359 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c32271-7a11-4546-92bf-2706c934fa12" containerName="maas-api" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805370 5914 cpu_manager.go:414] "containerMap: removing container" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805375 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805387 5914 cpu_manager.go:414] "containerMap: removing container" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" containerName="payload-processing" Oct 30 21:57:20.805390 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805391 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" containerName="payload-processing" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805479 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="360d0a39-dfaf-4624-bc7a-b57d598e216b" containerName="main" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805486 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef71a298-f0f5-4284-8059-245296585782" containerName="payload-processing" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805496 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c32271-7a11-4546-92bf-2706c934fa12" containerName="maas-api" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805503 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805513 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8eb883-8773-4f50-8f67-b3621be81239" containerName="payload-processing" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805532 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="0edca360-fbf2-4f5e-93dd-54e7c7910347" containerName="payload-pre-processing" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805538 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac64985c-d072-4a7b-b1a9-18733974a80b" containerName="payload-pre-processing" Oct 30 21:57:20.805854 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.805545 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bad27b-019f-4b10-b8d7-1a68b898992f" containerName="istio-proxy" Oct 30 21:57:20.808602 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.808587 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.882496 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.882463 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jdm6\" (UniqueName: \"kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.882659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.882604 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.882659 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.882641 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.882764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.882749 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.912901 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.912856 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dbe6da-serving-cert\"" Oct 30 21:57:20.915133 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.915110 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j"] Oct 30 21:57:20.983892 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.983841 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.984044 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.984004 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jdm6\" (UniqueName: \"kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.984113 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.984097 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.984152 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.984130 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.984251 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:20.984234 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-serving-cert: secret "maas-api-e2e-inf-b-dbe6da-serving-cert" not found Oct 30 21:57:20.984342 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:20.984330 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls podName:20589f2d-3051-4314-b71d-4f2fe21f5b7b nodeName:}" failed. No retries permitted until 2026-10-30 21:57:21.484296906 +0000 UTC m=+2393.386160963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls") pod "maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" (UID: "20589f2d-3051-4314-b71d-4f2fe21f5b7b") : secret "maas-api-e2e-inf-b-dbe6da-serving-cert" not found Oct 30 21:57:20.984828 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.984810 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:20.986436 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:20.986410 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:21.041708 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:21.041669 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jdm6\" (UniqueName: \"kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:21.490754 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:21.490721 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:21.493360 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:21.493332 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\") pod \"maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:21.721584 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:21.721549 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:22.097037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:22.097004 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j"] Oct 30 21:57:22.098596 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:57:22.098556 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20589f2d_3051_4314_b71d_4f2fe21f5b7b.slice/crio-ee2e7c3daa3613c3f08af252a95ffa3674df8fc95593d2673affe964ae8d56b3 WatchSource:0}: Error finding container ee2e7c3daa3613c3f08af252a95ffa3674df8fc95593d2673affe964ae8d56b3: Status 404 returned error can't find the container with id ee2e7c3daa3613c3f08af252a95ffa3674df8fc95593d2673affe964ae8d56b3 Oct 30 21:57:22.543927 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:22.543875 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" event={"ID":"20589f2d-3051-4314-b71d-4f2fe21f5b7b","Type":"ContainerStarted","Data":"ee2e7c3daa3613c3f08af252a95ffa3674df8fc95593d2673affe964ae8d56b3"} Oct 30 21:57:23.552779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:23.552739 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" event={"ID":"20589f2d-3051-4314-b71d-4f2fe21f5b7b","Type":"ContainerStarted","Data":"2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53"} Oct 30 21:57:23.553298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:23.553035 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:23.667521 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:23.667462 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" podStartSLOduration=2.818048561 podStartE2EDuration="3.667443179s" totalImagesPullingTime="849.394618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:23.666950692 +0000 UTC m=+2395.568814758" watchObservedRunningTime="2026-10-30 21:57:23.667443179 +0000 UTC m=+2395.569307246" Oct 30 21:57:28.644656 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:28.644615 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:57:29.236323 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:29.236292 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:57:29.239528 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:29.239503 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 21:57:29.564999 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:29.564924 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 21:57:41.628822 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.628786 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x"] Oct 30 21:57:41.629286 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.629030 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" containerID="cri-o://cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283" gracePeriod=30 Oct 30 21:57:41.916759 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.916729 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:57:41.941375 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.941344 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca\") pod \"6df0ea15-7904-44df-9d92-2751c2f9edd4\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " Oct 30 21:57:41.941543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.941416 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7\" (UniqueName: \"kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7\") pod \"6df0ea15-7904-44df-9d92-2751c2f9edd4\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " Oct 30 21:57:41.941543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.941451 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs\") pod \"6df0ea15-7904-44df-9d92-2751c2f9edd4\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " Oct 30 21:57:41.941543 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.941512 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\") pod \"6df0ea15-7904-44df-9d92-2751c2f9edd4\" (UID: \"6df0ea15-7904-44df-9d92-2751c2f9edd4\") " Oct 30 21:57:41.941780 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.941752 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca" pod "6df0ea15-7904-44df-9d92-2751c2f9edd4" (UID: "6df0ea15-7904-44df-9d92-2751c2f9edd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:41.942055 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.942037 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df0ea15-7904-44df-9d92-2751c2f9edd4-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:41.943810 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.943782 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls" pod "6df0ea15-7904-44df-9d92-2751c2f9edd4" (UID: "6df0ea15-7904-44df-9d92-2751c2f9edd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:41.943929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.943816 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs" pod "6df0ea15-7904-44df-9d92-2751c2f9edd4" (UID: "6df0ea15-7904-44df-9d92-2751c2f9edd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:41.943929 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:41.943924 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7" pod "6df0ea15-7904-44df-9d92-2751c2f9edd4" (UID: "6df0ea15-7904-44df-9d92-2751c2f9edd4"). InnerVolumeSpecName "kube-api-access-gppj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:42.043220 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.043182 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gppj7\" (UniqueName: \"kubernetes.io/projected/6df0ea15-7904-44df-9d92-2751c2f9edd4-kube-api-access-gppj7\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:42.043220 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.043219 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:42.043432 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.043232 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df0ea15-7904-44df-9d92-2751c2f9edd4-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:42.346340 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:42.346301 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:57:42.671525 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.671491 5914 generic.go:356] "Generic (PLEG): container finished" podID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerID="cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283" exitCode=0 Oct 30 21:57:42.672020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.671549 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" Oct 30 21:57:42.672020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.671556 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" event={"ID":"6df0ea15-7904-44df-9d92-2751c2f9edd4","Type":"ContainerDied","Data":"cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283"} Oct 30 21:57:42.672020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.671598 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x" event={"ID":"6df0ea15-7904-44df-9d92-2751c2f9edd4","Type":"ContainerDied","Data":"e48310eee33bc631e6355cd05593720bec5cd0444339582220242d342543b9e7"} Oct 30 21:57:42.672020 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.671621 5914 scope.go:122] "RemoveContainer" containerID="cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283" Oct 30 21:57:42.684491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.684451 5914 scope.go:122] "RemoveContainer" containerID="c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f" Oct 30 21:57:42.693337 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.693313 5914 scope.go:122] "RemoveContainer" containerID="cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283" Oct 30 21:57:42.693618 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:42.693595 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283\": container with ID starting with cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283 not found: ID does not exist" containerID="cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283" Oct 30 21:57:42.693696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.693631 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283"} err="failed to get container status \"cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283\": rpc error: code = NotFound desc = could not find container \"cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283\": container with ID starting with cbc94a292991bd348732941b1899ac06bcb2ae0515f06d46582a2477f1b32283 not found: ID does not exist" Oct 30 21:57:42.693696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.693656 5914 scope.go:122] "RemoveContainer" containerID="c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f" Oct 30 21:57:42.693974 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:42.693954 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f\": container with ID starting with c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f not found: ID does not exist" containerID="c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f" Oct 30 21:57:42.694036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.693980 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f"} err="failed to get container status \"c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f\": rpc error: code = NotFound desc = could not find container \"c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f\": container with ID starting with c19e268ca5d06e7ea4fa210156cb622d95e381359882d4088e584860b56fd07f not found: ID does not exist" Oct 30 21:57:42.836023 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.835990 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x"] Oct 30 21:57:42.885379 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:42.885344 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3b21b4-b6c6b66-l6g4x"] Oct 30 21:57:44.216460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:44.216418 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" path="/var/lib/kubelet/pods/6df0ea15-7904-44df-9d92-2751c2f9edd4/volumes" Oct 30 21:57:48.223713 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.223679 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l"] Oct 30 21:57:48.224251 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.223899 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" podUID="dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" containerName="maas-api" containerID="cri-o://3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8" gracePeriod=30 Oct 30 21:57:48.496274 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.496248 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:57:48.549340 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.549297 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca\") pod \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " Oct 30 21:57:48.549563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.549370 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm\" (UniqueName: \"kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm\") pod \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " Oct 30 21:57:48.549563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.549488 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs\") pod \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " Oct 30 21:57:48.549563 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.549538 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\") pod \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\" (UID: \"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66\") " Oct 30 21:57:48.549779 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.549751 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca" pod "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" (UID: "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:48.550114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.550090 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:48.551931 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.551902 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls" pod "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" (UID: "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:48.552342 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.552326 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs" pod "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" (UID: "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:48.552411 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.552396 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm" pod "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" (UID: "dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66"). InnerVolumeSpecName "kube-api-access-hwwzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:48.655909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.652707 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwwzm\" (UniqueName: \"kubernetes.io/projected/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-kube-api-access-hwwzm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:48.655909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.652770 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:48.655909 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.652786 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:48.719098 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.719059 5914 generic.go:356] "Generic (PLEG): container finished" podID="dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" containerID="3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8" exitCode=0 Oct 30 21:57:48.719318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.719137 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" Oct 30 21:57:48.719318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.719143 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" event={"ID":"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66","Type":"ContainerDied","Data":"3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8"} Oct 30 21:57:48.719318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.719205 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l" event={"ID":"dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66","Type":"ContainerDied","Data":"5ccb05d1d5fc7924e61aa7a49a91ec1df476924385cb6b2689d04144dfb74358"} Oct 30 21:57:48.719318 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.719229 5914 scope.go:122] "RemoveContainer" containerID="3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8" Oct 30 21:57:48.733114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.732496 5914 scope.go:122] "RemoveContainer" containerID="3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8" Oct 30 21:57:48.733114 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:57:48.733015 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8\": container with ID starting with 3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8 not found: ID does not exist" containerID="3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8" Oct 30 21:57:48.733114 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.733061 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8"} err="failed to get container status \"3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8\": rpc error: code = NotFound desc = could not find container \"3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8\": container with ID starting with 3eb647356da224e53b88eb3c2594f19395ba6e6c3c8ea7849bc98c59691a19c8 not found: ID does not exist" Oct 30 21:57:48.883906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.883552 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l"] Oct 30 21:57:48.886680 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:48.886647 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e4dadd-6bfcc57646-4dg6l"] Oct 30 21:57:50.216063 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:50.216017 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" path="/var/lib/kubelet/pods/dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66/volumes" Oct 30 21:57:57.373525 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:57.373488 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr"] Oct 30 21:57:57.374034 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:57.373693 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" containerID="cri-o://7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a" gracePeriod=30 Oct 30 21:57:58.796305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:58.796267 5914 patch_prober.go:29] interesting pod/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:58.796796 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:57:58.796335 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:58:01.213873 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:01.213842 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx"] Oct 30 21:58:01.214302 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:01.214037 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" containerID="cri-o://11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818" gracePeriod=30 Oct 30 21:58:02.642869 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.642802 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:58:02.714947 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.714910 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.714958 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715023 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715055 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715145 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715092 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715155 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715239 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715273 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715363 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715325 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token\") pod \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\" (UID: \"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4\") " Oct 30 21:58:02.715570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715369 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:02.715570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715449 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:02.715570 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715465 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:02.715713 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715596 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:02.715805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715782 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.715805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715809 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.716003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715819 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.716003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715827 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.716003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.715963 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:02.717339 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.717309 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:02.717439 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.717398 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:02.717771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.717749 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "kube-api-access-2jhml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:02.717839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.717792 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token" pod "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" (UID: "e79bd64b-0a3b-45bd-94bd-f5426fbf94c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:02.800032 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.799997 5914 generic.go:356] "Generic (PLEG): container finished" podID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerID="7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a" exitCode=0 Oct 30 21:58:02.800223 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.800041 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" Oct 30 21:58:02.800223 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.800077 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" event={"ID":"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4","Type":"ContainerDied","Data":"7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a"} Oct 30 21:58:02.800223 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.800143 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr" event={"ID":"e79bd64b-0a3b-45bd-94bd-f5426fbf94c4","Type":"ContainerDied","Data":"6f38af54c9d7e685e5fdfea3fcce05fa1e589d7d68bac8df839d3df40e5cda27"} Oct 30 21:58:02.800223 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.800174 5914 scope.go:122] "RemoveContainer" containerID="7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a" Oct 30 21:58:02.811504 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.811457 5914 scope.go:122] "RemoveContainer" containerID="9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a" Oct 30 21:58:02.817671 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.817638 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jhml\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-kube-api-access-2jhml\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.817808 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.817679 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.817808 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.817694 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.817808 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.817708 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.817808 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.817723 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:02.822135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.822112 5914 scope.go:122] "RemoveContainer" containerID="7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a" Oct 30 21:58:02.822453 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:02.822432 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a\": container with ID starting with 7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a not found: ID does not exist" containerID="7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a" Oct 30 21:58:02.822501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.822465 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a"} err="failed to get container status \"7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a\": rpc error: code = NotFound desc = could not find container \"7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a\": container with ID starting with 7d39806fa2f1af5cbfea76542c9789833e53e4465cb8880c6baba6c666e4f40a not found: ID does not exist" Oct 30 21:58:02.822501 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.822483 5914 scope.go:122] "RemoveContainer" containerID="9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a" Oct 30 21:58:02.822789 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:02.822765 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a\": container with ID starting with 9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a not found: ID does not exist" containerID="9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a" Oct 30 21:58:02.822851 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.822799 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a"} err="failed to get container status \"9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a\": rpc error: code = NotFound desc = could not find container \"9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a\": container with ID starting with 9ecbabd8f3bc71282a8f9a3e8631f79cc789e9abe9ab42944695a5d6fce4960a not found: ID does not exist" Oct 30 21:58:02.963069 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.962984 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr"] Oct 30 21:58:02.966900 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:02.966863 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e4dadd-openshift-default-555c77774c-tc7zr"] Oct 30 21:58:04.216839 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:04.216806 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" path="/var/lib/kubelet/pods/e79bd64b-0a3b-45bd-94bd-f5426fbf94c4/volumes" Oct 30 21:58:05.132423 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:05.132386 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:58:06.481513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.481454 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:58:06.557701 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.557671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.557856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.557726 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.557856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.557752 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.557856 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.557796 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.558039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.557874 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.558039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.557955 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.558039 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558017 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.558194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558042 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.558194 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558087 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data\") pod \"81bbf7f4-758d-497d-9bfe-a4273c871f63\" (UID: \"81bbf7f4-758d-497d-9bfe-a4273c871f63\") " Oct 30 21:58:06.558387 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558301 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:06.558526 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558449 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.558597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558534 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.558760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558696 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.558760 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558722 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.558874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558822 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81bbf7f4-758d-497d-9bfe-a4273c871f63-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.558874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558844 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.558874 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.558862 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.560244 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.560220 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:06.560349 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.560254 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "kube-api-access-84jz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:06.560630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.560605 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:06.560630 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.560617 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token" pod "81bbf7f4-758d-497d-9bfe-a4273c871f63" (UID: "81bbf7f4-758d-497d-9bfe-a4273c871f63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:06.660015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.659977 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.660015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.660007 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.660015 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.660020 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.660258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.660032 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81bbf7f4-758d-497d-9bfe-a4273c871f63-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.660258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.660044 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84jz4\" (UniqueName: \"kubernetes.io/projected/81bbf7f4-758d-497d-9bfe-a4273c871f63-kube-api-access-84jz4\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.660258 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.660058 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81bbf7f4-758d-497d-9bfe-a4273c871f63-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:06.826620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.826533 5914 generic.go:356] "Generic (PLEG): container finished" podID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerID="11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818" exitCode=0 Oct 30 21:58:06.826620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.826587 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" Oct 30 21:58:06.826620 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.826610 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" event={"ID":"81bbf7f4-758d-497d-9bfe-a4273c871f63","Type":"ContainerDied","Data":"11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818"} Oct 30 21:58:06.826847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.826669 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx" event={"ID":"81bbf7f4-758d-497d-9bfe-a4273c871f63","Type":"ContainerDied","Data":"eebc189db4ac603ff4822fb031c68426abcfd4eed638949bd7d3062de1202c1d"} Oct 30 21:58:06.826847 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.826700 5914 scope.go:122] "RemoveContainer" containerID="11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818" Oct 30 21:58:06.841060 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.840327 5914 scope.go:122] "RemoveContainer" containerID="04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b" Oct 30 21:58:06.849488 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.849466 5914 scope.go:122] "RemoveContainer" containerID="11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818" Oct 30 21:58:06.849747 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:06.849723 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818\": container with ID starting with 11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818 not found: ID does not exist" containerID="11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818" Oct 30 21:58:06.849826 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.849760 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818"} err="failed to get container status \"11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818\": rpc error: code = NotFound desc = could not find container \"11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818\": container with ID starting with 11790fc65f70b6f1b8402b910ebd0e8317457acf6c9da6232500fa116c41e818 not found: ID does not exist" Oct 30 21:58:06.849826 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.849787 5914 scope.go:122] "RemoveContainer" containerID="04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b" Oct 30 21:58:06.850033 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:06.850014 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b\": container with ID starting with 04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b not found: ID does not exist" containerID="04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b" Oct 30 21:58:06.850086 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.850039 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b"} err="failed to get container status \"04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b\": rpc error: code = NotFound desc = could not find container \"04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b\": container with ID starting with 04e9d04c9f0efb67991f3482815958d8bf9be980ef8d053e81e473edf282c10b not found: ID does not exist" Oct 30 21:58:06.990722 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.990693 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx"] Oct 30 21:58:06.995327 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:06.995298 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-3b21b4-openshift-default-9cc79755f-s4frx"] Oct 30 21:58:08.217135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:08.217047 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" path="/var/lib/kubelet/pods/81bbf7f4-758d-497d-9bfe-a4273c871f63/volumes" Oct 30 21:58:12.345694 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:12.345653 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:58:22.083955 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.083910 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg"] Oct 30 21:58:22.084670 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084646 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084674 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084696 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084703 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084729 5914 cpu_manager.go:414] "containerMap: removing container" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084737 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084753 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084761 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084791 5914 cpu_manager.go:414] "containerMap: removing container" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084798 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" Oct 30 21:58:22.084823 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084827 5914 cpu_manager.go:414] "containerMap: removing container" podUID="dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" containerName="maas-api" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084834 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" containerName="maas-api" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.084992 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce0c4c5-9e4c-4d0c-869c-f8105f3b6a66" containerName="maas-api" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.085009 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.085022 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.085033 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.085045 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" Oct 30 21:58:22.085460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.085053 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df0ea15-7904-44df-9d92-2751c2f9edd4" containerName="maas-api" Oct 30 21:58:22.088533 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.088513 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.193009 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.192974 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e9aa8e\"/\"openshift-service-ca.crt\"" Oct 30 21:58:22.193559 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.193537 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e9aa8e\"/\"kube-root-ca.crt\"" Oct 30 21:58:22.193631 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.193612 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e9aa8e\"/\"default-dockercfg-ghxbf\"" Oct 30 21:58:22.193737 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.193716 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e9aa8e\"/\"test-model-e9aa8e-kserve-self-signed-certs\"" Oct 30 21:58:22.194981 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.194959 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg"] Oct 30 21:58:22.238490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.237326 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbxt\" (UniqueName: \"kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.238490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.237557 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.238490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.237593 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.238490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.237689 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.238490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.237811 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.238490 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.237871 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.338920 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339018 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.338985 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbxt\" (UniqueName: \"kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339098 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339126 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339167 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339270 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339266 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339510 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339394 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339574 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339535 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.339693 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.339590 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.341557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.341537 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.341696 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.341680 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.396801 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.396767 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbxt\" (UniqueName: \"kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt\") pod \"test-model-e9aa8e-kserve-65b99bb8cc-44wwg\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.400965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.400939 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:22.776970 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.776935 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg"] Oct 30 21:58:22.778170 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:58:22.778143 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd615093e_57ba_466f_9d49_1825cc8c7dee.slice/crio-9f101b1afa72c5970f0331f8cddf2c7463e699609849e0da6febd15491d21b6f WatchSource:0}: Error finding container 9f101b1afa72c5970f0331f8cddf2c7463e699609849e0da6febd15491d21b6f: Status 404 returned error can't find the container with id 9f101b1afa72c5970f0331f8cddf2c7463e699609849e0da6febd15491d21b6f Oct 30 21:58:22.919962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.919918 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" event={"ID":"d615093e-57ba-466f-9d49-1825cc8c7dee","Type":"ContainerStarted","Data":"8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578"} Oct 30 21:58:22.920138 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:22.919969 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" event={"ID":"d615093e-57ba-466f-9d49-1825cc8c7dee","Type":"ContainerStarted","Data":"9f101b1afa72c5970f0331f8cddf2c7463e699609849e0da6febd15491d21b6f"} Oct 30 21:58:23.033447 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:23.033323 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" podStartSLOduration=1.033306701 podStartE2EDuration="1.033306701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.03286056 +0000 UTC m=+2454.934724625" watchObservedRunningTime="2026-10-30 21:58:23.033306701 +0000 UTC m=+2454.935170783" Oct 30 21:58:23.401962 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:23.401930 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:32.372783 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:32.372751 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:58:32.401389 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:32.401359 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:32.414541 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:32.414518 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:32.985303 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:32.985273 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 21:58:42.346103 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:42.346064 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:58:51.458087 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.458048 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t"] Oct 30 21:58:51.460386 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.458238 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" podUID="a1de05e3-8a4c-4900-a123-0f83a2fd06aa" containerName="maas-api" containerID="cri-o://bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be" gracePeriod=30 Oct 30 21:58:51.723519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.723455 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:58:51.879868 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.879830 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs\") pod \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " Oct 30 21:58:51.880080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.879898 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\") pod \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " Oct 30 21:58:51.880080 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.880063 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh\" (UniqueName: \"kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh\") pod \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " Oct 30 21:58:51.880201 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.880097 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca\") pod \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\" (UID: \"a1de05e3-8a4c-4900-a123-0f83a2fd06aa\") " Oct 30 21:58:51.880567 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.880530 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca" pod "a1de05e3-8a4c-4900-a123-0f83a2fd06aa" (UID: "a1de05e3-8a4c-4900-a123-0f83a2fd06aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:51.882372 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.882345 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls" pod "a1de05e3-8a4c-4900-a123-0f83a2fd06aa" (UID: "a1de05e3-8a4c-4900-a123-0f83a2fd06aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:51.882470 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.882368 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs" pod "a1de05e3-8a4c-4900-a123-0f83a2fd06aa" (UID: "a1de05e3-8a4c-4900-a123-0f83a2fd06aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:51.882470 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.882439 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh" pod "a1de05e3-8a4c-4900-a123-0f83a2fd06aa" (UID: "a1de05e3-8a4c-4900-a123-0f83a2fd06aa"). InnerVolumeSpecName "kube-api-access-49fsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:51.981519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.981426 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49fsh\" (UniqueName: \"kubernetes.io/projected/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-kube-api-access-49fsh\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:51.981519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.981461 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:51.981519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.981474 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:51.981519 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:51.981488 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1de05e3-8a4c-4900-a123-0f83a2fd06aa-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.086554 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.086521 5914 generic.go:356] "Generic (PLEG): container finished" podID="a1de05e3-8a4c-4900-a123-0f83a2fd06aa" containerID="bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be" exitCode=0 Oct 30 21:58:52.086750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.086564 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" Oct 30 21:58:52.086750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.086615 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" event={"ID":"a1de05e3-8a4c-4900-a123-0f83a2fd06aa","Type":"ContainerDied","Data":"bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be"} Oct 30 21:58:52.086750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.086709 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t" event={"ID":"a1de05e3-8a4c-4900-a123-0f83a2fd06aa","Type":"ContainerDied","Data":"62d417a50cb5ad93e624d7fd2ea8496d08c555ff7443766afe867b5d1dfc481d"} Oct 30 21:58:52.086750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.086737 5914 scope.go:122] "RemoveContainer" containerID="bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be" Oct 30 21:58:52.096768 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.096745 5914 scope.go:122] "RemoveContainer" containerID="bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be" Oct 30 21:58:52.097101 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:58:52.097080 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be\": container with ID starting with bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be not found: ID does not exist" containerID="bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be" Oct 30 21:58:52.097203 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.097109 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be"} err="failed to get container status \"bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be\": rpc error: code = NotFound desc = could not find container \"bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be\": container with ID starting with bc1d3ae4ed8d0804ca1a80e9307fa8da0467740b81c9f71b317383975a29a8be not found: ID does not exist" Oct 30 21:58:52.249334 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.249242 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t"] Oct 30 21:58:52.250932 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:52.250910 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d6a241-cf9d6f847-t527t"] Oct 30 21:58:54.217416 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:58:54.217376 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1de05e3-8a4c-4900-a123-0f83a2fd06aa" path="/var/lib/kubelet/pods/a1de05e3-8a4c-4900-a123-0f83a2fd06aa/volumes" Oct 30 21:59:00.660298 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660264 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv"] Oct 30 21:59:00.660842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660739 5914 cpu_manager.go:414] "containerMap: removing container" podUID="a1de05e3-8a4c-4900-a123-0f83a2fd06aa" containerName="maas-api" Oct 30 21:59:00.660842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660750 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1de05e3-8a4c-4900-a123-0f83a2fd06aa" containerName="maas-api" Oct 30 21:59:00.660842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660793 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" Oct 30 21:59:00.660842 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660798 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79bd64b-0a3b-45bd-94bd-f5426fbf94c4" containerName="istio-proxy" Oct 30 21:59:00.661065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660858 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1de05e3-8a4c-4900-a123-0f83a2fd06aa" containerName="maas-api" Oct 30 21:59:00.661065 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.660870 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bbf7f4-758d-497d-9bfe-a4273c871f63" containerName="istio-proxy" Oct 30 21:59:00.665597 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.665578 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.764813 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.764775 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.765001 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.764981 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.765050 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.765011 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.765050 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.765039 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.765131 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.765109 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4pg\" (UniqueName: \"kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.765176 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.765164 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.767558 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.767538 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dbe6da\"/\"kube-root-ca.crt\"" Oct 30 21:59:00.767638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.767539 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dbe6da\"/\"openshift-service-ca.crt\"" Oct 30 21:59:00.767638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.767564 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dbe6da\"/\"test-model-dbe6da-kserve-self-signed-certs\"" Oct 30 21:59:00.767638 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.767538 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dbe6da\"/\"default-dockercfg-z6plm\"" Oct 30 21:59:00.781705 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.781681 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv"] Oct 30 21:59:00.865674 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.865637 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.865911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.865747 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.865911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.865772 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.865911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.865793 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.865911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.865812 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4pg\" (UniqueName: \"kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.865911 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.865859 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.866603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.866515 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.866603 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.866559 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.866860 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.866787 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.868600 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.868575 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.868707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.868580 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.927712 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.927630 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4pg\" (UniqueName: \"kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg\") pod \"test-model-dbe6da-kserve-5ff89c556c-28jmv\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:00.977557 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:00.977521 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:01.356916 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:01.356859 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv"] Oct 30 21:59:02.143580 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:02.143537 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" event={"ID":"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52","Type":"ContainerStarted","Data":"0c79c63ad6744f9f989d937fb184b51b23370186d30f15b727dbb8e5ca969fef"} Oct 30 21:59:02.144048 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:02.143592 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" event={"ID":"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52","Type":"ContainerStarted","Data":"aff5391c561edd98375176c7fb65eca63d21db3b1eb26aee04fc3b58861d8b32"} Oct 30 21:59:02.259284 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:02.259227 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" podStartSLOduration=2.259210513 podStartE2EDuration="2.259210513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:02.258336104 +0000 UTC m=+2494.160200167" watchObservedRunningTime="2026-10-30 21:59:02.259210513 +0000 UTC m=+2494.161074574" Oct 30 21:59:02.940450 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:02.940415 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:59:02.978153 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:02.978120 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:06.121067 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:06.121030 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz"] Oct 30 21:59:06.176752 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:06.174215 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb"] Oct 30 21:59:06.177805 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:06.177773 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n"] Oct 30 21:59:10.977767 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:10.977732 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:10.990969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:10.990943 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:11.202395 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:11.202366 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 21:59:12.346166 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:12.346130 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:59:13.344512 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:13.344473 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb"] Oct 30 21:59:18.344844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:18.344785 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" podUID="457f7406-0588-4389-9731-550c7ae5f988" containerName="payload-pre-processing" containerID="cri-o://36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0" gracePeriod=25 Oct 30 21:59:21.142643 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.142604 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" containerID="cri-o://a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef" gracePeriod=45 Oct 30 21:59:21.203794 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.203751 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" containerID="cri-o://a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756" gracePeriod=45 Oct 30 21:59:21.203967 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.203898 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" containerID="cri-o://f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7" gracePeriod=45 Oct 30 21:59:21.467356 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.467325 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:59:21.492634 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.492611 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:59:21.501686 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.501652 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:59:21.601930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.601800 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm\") pod \"570faa65-6036-4f69-b260-832f77f691d4\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " Oct 30 21:59:21.601930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.601852 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9\" (UniqueName: \"kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9\") pod \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " Oct 30 21:59:21.601930 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.601909 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir\") pod \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.601940 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home\") pod \"39953fdd-297a-4bc0-af16-9d4d9b68170a\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602016 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home\") pod \"570faa65-6036-4f69-b260-832f77f691d4\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602034 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home\") pod \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602056 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm\") pod \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602078 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache\") pod \"570faa65-6036-4f69-b260-832f77f691d4\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602105 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz\" (UniqueName: \"kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz\") pod \"39953fdd-297a-4bc0-af16-9d4d9b68170a\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602131 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache\") pod \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " Oct 30 21:59:21.602216 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602161 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir\") pod \"570faa65-6036-4f69-b260-832f77f691d4\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602244 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs\" (UniqueName: \"kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs\") pod \"39953fdd-297a-4bc0-af16-9d4d9b68170a\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602263 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm\") pod \"39953fdd-297a-4bc0-af16-9d4d9b68170a\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602299 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir\") pod \"39953fdd-297a-4bc0-af16-9d4d9b68170a\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602330 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs\") pod \"570faa65-6036-4f69-b260-832f77f691d4\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602371 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg\" (UniqueName: \"kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg\") pod \"570faa65-6036-4f69-b260-832f77f691d4\" (UID: \"570faa65-6036-4f69-b260-832f77f691d4\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602397 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache\") pod \"39953fdd-297a-4bc0-af16-9d4d9b68170a\" (UID: \"39953fdd-297a-4bc0-af16-9d4d9b68170a\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602417 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs\") pod \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\" (UID: \"ad107193-e9ed-4053-8855-d8aa7e53d9f6\") " Oct 30 21:59:21.602646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602437 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home" pod "570faa65-6036-4f69-b260-832f77f691d4" (UID: "570faa65-6036-4f69-b260-832f77f691d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.603028 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.602712 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.603764 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.603518 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache" pod "39953fdd-297a-4bc0-af16-9d4d9b68170a" (UID: "39953fdd-297a-4bc0-af16-9d4d9b68170a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.603917 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.603805 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir" pod "ad107193-e9ed-4053-8855-d8aa7e53d9f6" (UID: "ad107193-e9ed-4053-8855-d8aa7e53d9f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.604003 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.603919 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache" pod "570faa65-6036-4f69-b260-832f77f691d4" (UID: "570faa65-6036-4f69-b260-832f77f691d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.604272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.604117 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir" pod "39953fdd-297a-4bc0-af16-9d4d9b68170a" (UID: "39953fdd-297a-4bc0-af16-9d4d9b68170a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.604272 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.604142 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home" pod "ad107193-e9ed-4053-8855-d8aa7e53d9f6" (UID: "ad107193-e9ed-4053-8855-d8aa7e53d9f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.604866 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.604575 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache" pod "ad107193-e9ed-4053-8855-d8aa7e53d9f6" (UID: "ad107193-e9ed-4053-8855-d8aa7e53d9f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.605415 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.605378 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home" pod "39953fdd-297a-4bc0-af16-9d4d9b68170a" (UID: "39953fdd-297a-4bc0-af16-9d4d9b68170a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.605506 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.605454 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm" pod "570faa65-6036-4f69-b260-832f77f691d4" (UID: "570faa65-6036-4f69-b260-832f77f691d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.605665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.605642 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir" pod "570faa65-6036-4f69-b260-832f77f691d4" (UID: "570faa65-6036-4f69-b260-832f77f691d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.606126 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.606080 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs" pod "39953fdd-297a-4bc0-af16-9d4d9b68170a" (UID: "39953fdd-297a-4bc0-af16-9d4d9b68170a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:21.606750 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.606722 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm" pod "ad107193-e9ed-4053-8855-d8aa7e53d9f6" (UID: "ad107193-e9ed-4053-8855-d8aa7e53d9f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.606989 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.606965 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9" pod "ad107193-e9ed-4053-8855-d8aa7e53d9f6" (UID: "ad107193-e9ed-4053-8855-d8aa7e53d9f6"). InnerVolumeSpecName "kube-api-access-kbrr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:21.607359 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.607334 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm" pod "39953fdd-297a-4bc0-af16-9d4d9b68170a" (UID: "39953fdd-297a-4bc0-af16-9d4d9b68170a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:59:21.607748 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.607721 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs" pod "570faa65-6036-4f69-b260-832f77f691d4" (UID: "570faa65-6036-4f69-b260-832f77f691d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:21.608066 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.608044 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg" pod "570faa65-6036-4f69-b260-832f77f691d4" (UID: "570faa65-6036-4f69-b260-832f77f691d4"). InnerVolumeSpecName "kube-api-access-8fncg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:21.608649 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.608619 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz" pod "39953fdd-297a-4bc0-af16-9d4d9b68170a" (UID: "39953fdd-297a-4bc0-af16-9d4d9b68170a"). InnerVolumeSpecName "kube-api-access-xbsmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:21.609478 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.609454 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs" pod "ad107193-e9ed-4053-8855-d8aa7e53d9f6" (UID: "ad107193-e9ed-4053-8855-d8aa7e53d9f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:21.704364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704291 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704323 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704332 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704341 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704349 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704360 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsmz\" (UniqueName: \"kubernetes.io/projected/39953fdd-297a-4bc0-af16-9d4d9b68170a-kube-api-access-xbsmz\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704370 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad107193-e9ed-4053-8855-d8aa7e53d9f6-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704379 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704387 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39953fdd-297a-4bc0-af16-9d4d9b68170a-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704395 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704405 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704412 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570faa65-6036-4f69-b260-832f77f691d4-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704420 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fncg\" (UniqueName: \"kubernetes.io/projected/570faa65-6036-4f69-b260-832f77f691d4-kube-api-access-8fncg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704428 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39953fdd-297a-4bc0-af16-9d4d9b68170a-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704436 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad107193-e9ed-4053-8855-d8aa7e53d9f6-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704444 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570faa65-6036-4f69-b260-832f77f691d4-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:21.704665 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:21.704452 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrr9\" (UniqueName: \"kubernetes.io/projected/ad107193-e9ed-4053-8855-d8aa7e53d9f6-kube-api-access-kbrr9\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:22.257904 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.257855 5914 generic.go:356] "Generic (PLEG): container finished" podID="570faa65-6036-4f69-b260-832f77f691d4" containerID="f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7" exitCode=0 Oct 30 21:59:22.258367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.257922 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" Oct 30 21:59:22.258367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.257921 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" event={"ID":"570faa65-6036-4f69-b260-832f77f691d4","Type":"ContainerDied","Data":"f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7"} Oct 30 21:59:22.258367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.258054 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" event={"ID":"570faa65-6036-4f69-b260-832f77f691d4","Type":"ContainerDied","Data":"f2f98e93a920ff93aab1ffe7f7e60450bdd9faff3f4ba3556d11a928b6d000ab"} Oct 30 21:59:22.258367 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.258080 5914 scope.go:122] "RemoveContainer" containerID="f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7" Oct 30 21:59:22.259445 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.259425 5914 generic.go:356] "Generic (PLEG): container finished" podID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerID="a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756" exitCode=0 Oct 30 21:59:22.259547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.259481 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" Oct 30 21:59:22.259547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.259498 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" event={"ID":"39953fdd-297a-4bc0-af16-9d4d9b68170a","Type":"ContainerDied","Data":"a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756"} Oct 30 21:59:22.259547 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.259527 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n" event={"ID":"39953fdd-297a-4bc0-af16-9d4d9b68170a","Type":"ContainerDied","Data":"ec7a9c1f1cda82089989a3b9dffd486d8bb8dae90935fb361323056e68269e8d"} Oct 30 21:59:22.261452 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.261409 5914 generic.go:356] "Generic (PLEG): container finished" podID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerID="a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef" exitCode=0 Oct 30 21:59:22.261610 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.261473 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" event={"ID":"ad107193-e9ed-4053-8855-d8aa7e53d9f6","Type":"ContainerDied","Data":"a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef"} Oct 30 21:59:22.261610 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.261497 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" event={"ID":"ad107193-e9ed-4053-8855-d8aa7e53d9f6","Type":"ContainerDied","Data":"08c377a979a12662a1a3fb5893c367a7cb54b767f15699b3c9567a7a9194b4e8"} Oct 30 21:59:22.261610 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.261511 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz" Oct 30 21:59:22.269117 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.269097 5914 scope.go:122] "RemoveContainer" containerID="f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7" Oct 30 21:59:22.269431 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:22.269401 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7\": container with ID starting with f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7 not found: ID does not exist" containerID="f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7" Oct 30 21:59:22.269513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.269438 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7"} err="failed to get container status \"f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7\": rpc error: code = NotFound desc = could not find container \"f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7\": container with ID starting with f430b4867c83dcc99b25449c628afe04bcb45fe12af26c4b19e88ae9c9cbb0f7 not found: ID does not exist" Oct 30 21:59:22.269513 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.269455 5914 scope.go:122] "RemoveContainer" containerID="a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756" Oct 30 21:59:22.279497 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.279472 5914 scope.go:122] "RemoveContainer" containerID="a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756" Oct 30 21:59:22.279818 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:22.279796 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756\": container with ID starting with a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756 not found: ID does not exist" containerID="a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756" Oct 30 21:59:22.279937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.279825 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756"} err="failed to get container status \"a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756\": rpc error: code = NotFound desc = could not find container \"a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756\": container with ID starting with a22887f9a892ff3fbfea60129427b9ac374128633d313d97a5b4394cb2400756 not found: ID does not exist" Oct 30 21:59:22.279937 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.279845 5914 scope.go:122] "RemoveContainer" containerID="a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef" Oct 30 21:59:22.289255 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.289231 5914 scope.go:122] "RemoveContainer" containerID="a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef" Oct 30 21:59:22.289526 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:22.289501 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef\": container with ID starting with a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef not found: ID does not exist" containerID="a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef" Oct 30 21:59:22.289616 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.289534 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef"} err="failed to get container status \"a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef\": rpc error: code = NotFound desc = could not find container \"a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef\": container with ID starting with a11bd0ec068f9b689392dcf408d0ab895cbc183e7593597515ed4ef916c2d2ef not found: ID does not exist" Oct 30 21:59:22.404049 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.404001 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:22.420208 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.420121 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz"] Oct 30 21:59:22.424991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.424966 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-2-w0-1b5b5f-kserve-8989d55b5-7rjwz"] Oct 30 21:59:22.538821 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.538783 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb"] Oct 30 21:59:22.540364 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.540344 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/distinct-w0-1b5b5f-kserve-b5b96f77d-65xsb"] Oct 30 21:59:22.648268 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.648237 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n"] Oct 30 21:59:22.653500 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:22.653472 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1b5b5f/simulator-w0-1b5b5f-kserve-99ff96fd6-vtw5n"] Oct 30 21:59:24.216036 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:24.216006 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" path="/var/lib/kubelet/pods/39953fdd-297a-4bc0-af16-9d4d9b68170a/volumes" Oct 30 21:59:24.216460 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:24.216446 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570faa65-6036-4f69-b260-832f77f691d4" path="/var/lib/kubelet/pods/570faa65-6036-4f69-b260-832f77f691d4/volumes" Oct 30 21:59:24.216819 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:24.216807 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" path="/var/lib/kubelet/pods/ad107193-e9ed-4053-8855-d8aa7e53d9f6/volumes" Oct 30 21:59:25.232910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:25.232855 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:59:26.622458 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:26.622417 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" podUID="457f7406-0588-4389-9731-550c7ae5f988" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:59:27.216806 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.216773 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7301-73701a"] Oct 30 21:59:27.217319 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217304 5914 cpu_manager.go:414] "containerMap: removing container" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" Oct 30 21:59:27.217369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217321 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" Oct 30 21:59:27.217369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217335 5914 cpu_manager.go:414] "containerMap: removing container" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" Oct 30 21:59:27.217369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217340 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" Oct 30 21:59:27.217369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217356 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" Oct 30 21:59:27.217369 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217365 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" Oct 30 21:59:27.217545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217478 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="570faa65-6036-4f69-b260-832f77f691d4" containerName="main" Oct 30 21:59:27.217545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217487 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="39953fdd-297a-4bc0-af16-9d4d9b68170a" containerName="main" Oct 30 21:59:27.217545 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.217494 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad107193-e9ed-4053-8855-d8aa7e53d9f6" containerName="main" Oct 30 21:59:27.222365 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.222347 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:27.275454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.275416 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m2hp5\"" Oct 30 21:59:27.326965 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.326928 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7301-73701a"] Oct 30 21:59:27.371869 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.371826 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nj74\" (UniqueName: \"kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74\") pod \"test-curl-7301-73701a\" (UID: \"377e5e8e-879f-422f-b112-8393cae3a2a7\") " pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:27.473948 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.473850 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nj74\" (UniqueName: \"kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74\") pod \"test-curl-7301-73701a\" (UID: \"377e5e8e-879f-422f-b112-8393cae3a2a7\") " pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:27.532326 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.532286 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nj74\" (UniqueName: \"kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74\") pod \"test-curl-7301-73701a\" (UID: \"377e5e8e-879f-422f-b112-8393cae3a2a7\") " pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:27.534171 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.534145 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:27.916397 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.916372 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7301-73701a"] Oct 30 21:59:27.918798 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:59:27.918767 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod377e5e8e_879f_422f_b112_8393cae3a2a7.slice/crio-5d11bacb37cdcd17f6605604f5529702554389903fb60a4e6089d7d9b4bd8e91 WatchSource:0}: Error finding container 5d11bacb37cdcd17f6605604f5529702554389903fb60a4e6089d7d9b4bd8e91: Status 404 returned error can't find the container with id 5d11bacb37cdcd17f6605604f5529702554389903fb60a4e6089d7d9b4bd8e91 Oct 30 21:59:27.920771 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:27.920754 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:59:28.296829 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:28.296745 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7301-73701a" event={"ID":"377e5e8e-879f-422f-b112-8393cae3a2a7","Type":"ContainerStarted","Data":"5d11bacb37cdcd17f6605604f5529702554389903fb60a4e6089d7d9b4bd8e91"} Oct 30 21:59:35.333385 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:35.333292 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7301-73701a" event={"ID":"377e5e8e-879f-422f-b112-8393cae3a2a7","Type":"ContainerStarted","Data":"7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637"} Oct 30 21:59:35.449135 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:35.449077 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7301-73701a" podStartSLOduration=1.323386756 podStartE2EDuration="8.44906236s" totalImagesPullingTime="7.125675604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:35.448915695 +0000 UTC m=+2527.350779757" watchObservedRunningTime="2026-10-30 21:59:35.44906236 +0000 UTC m=+2527.350926424" Oct 30 21:59:36.622183 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:36.622098 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" podUID="457f7406-0588-4389-9731-550c7ae5f988" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:59:36.958906 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:36.958849 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7301-73701a"] Oct 30 21:59:36.963511 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:36.963482 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7301-73701a"] Oct 30 21:59:37.343844 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:37.343759 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7301-73701a" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" containerName="test-curl-7301-73701a" containerID="cri-o://7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637" gracePeriod=2 Oct 30 21:59:37.395210 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:37.395170 5914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7301-73701a\" is forbidden: User \"system:node:ip-10-0-142-119.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-119.us-west-1.compute.internal' and this object" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:38.263446 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:38.263405 5914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7301-73701a\" is forbidden: User \"system:node:ip-10-0-142-119.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-119.us-west-1.compute.internal' and this object" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:38.609162 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.609132 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:59:38.706029 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.705990 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume\") pod \"457f7406-0588-4389-9731-550c7ae5f988\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " Oct 30 21:59:38.706245 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.706077 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz\" (UniqueName: \"kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz\") pod \"457f7406-0588-4389-9731-550c7ae5f988\" (UID: \"457f7406-0588-4389-9731-550c7ae5f988\") " Oct 30 21:59:38.706397 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.706373 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume" pod "457f7406-0588-4389-9731-550c7ae5f988" (UID: "457f7406-0588-4389-9731-550c7ae5f988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:38.706542 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.706512 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457f7406-0588-4389-9731-550c7ae5f988-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:38.708358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.708340 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz" pod "457f7406-0588-4389-9731-550c7ae5f988" (UID: "457f7406-0588-4389-9731-550c7ae5f988"). InnerVolumeSpecName "kube-api-access-4r8dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:38.807969 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:38.807934 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r8dz\" (UniqueName: \"kubernetes.io/projected/457f7406-0588-4389-9731-550c7ae5f988-kube-api-access-4r8dz\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:39.364732 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.364696 5914 generic.go:356] "Generic (PLEG): container finished" podID="457f7406-0588-4389-9731-550c7ae5f988" containerID="36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0" exitCode=0 Oct 30 21:59:39.365165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.364775 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" Oct 30 21:59:39.365165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.364777 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" event={"ID":"457f7406-0588-4389-9731-550c7ae5f988","Type":"ContainerDied","Data":"36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0"} Oct 30 21:59:39.365165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.364940 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb" event={"ID":"457f7406-0588-4389-9731-550c7ae5f988","Type":"ContainerDied","Data":"84990d09fb394c2ae83f05c95bc6e4244dd6d03cb9c1fbe085a64d50ba0040d0"} Oct 30 21:59:39.365165 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.364965 5914 scope.go:122] "RemoveContainer" containerID="36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0" Oct 30 21:59:39.408037 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.408008 5914 scope.go:122] "RemoveContainer" containerID="36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0" Oct 30 21:59:39.409322 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:39.409286 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0\": container with ID starting with 36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0 not found: ID does not exist" containerID="36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0" Oct 30 21:59:39.409480 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.409335 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0"} err="failed to get container status \"36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0\": rpc error: code = NotFound desc = could not find container \"36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0\": container with ID starting with 36259eff79ea9d85e13b7c5e7f9dfe7c206c4d7271c0249d0cd19e435b08c7d0 not found: ID does not exist" Oct 30 21:59:39.529111 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.529080 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb"] Oct 30 21:59:39.578241 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.578210 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1b5b5f-5bd94f6ddc-8g4nb"] Oct 30 21:59:39.579723 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.579696 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:39.617313 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.617233 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74\" (UniqueName: \"kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74\") pod \"377e5e8e-879f-422f-b112-8393cae3a2a7\" (UID: \"377e5e8e-879f-422f-b112-8393cae3a2a7\") " Oct 30 21:59:39.619451 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.619423 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74" pod "377e5e8e-879f-422f-b112-8393cae3a2a7" (UID: "377e5e8e-879f-422f-b112-8393cae3a2a7"). InnerVolumeSpecName "kube-api-access-6nj74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:39.718957 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:39.718923 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nj74\" (UniqueName: \"kubernetes.io/projected/377e5e8e-879f-422f-b112-8393cae3a2a7-kube-api-access-6nj74\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:40.215907 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.215853 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" path="/var/lib/kubelet/pods/377e5e8e-879f-422f-b112-8393cae3a2a7/volumes" Oct 30 21:59:40.216286 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.216271 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457f7406-0588-4389-9731-550c7ae5f988" path="/var/lib/kubelet/pods/457f7406-0588-4389-9731-550c7ae5f988/volumes" Oct 30 21:59:40.372725 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.372689 5914 generic.go:356] "Generic (PLEG): container finished" podID="377e5e8e-879f-422f-b112-8393cae3a2a7" containerID="7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637" exitCode=137 Oct 30 21:59:40.373184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.372743 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-73701a" Oct 30 21:59:40.373184 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.372780 5914 scope.go:122] "RemoveContainer" containerID="7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637" Oct 30 21:59:40.381497 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.381479 5914 scope.go:122] "RemoveContainer" containerID="7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637" Oct 30 21:59:40.381771 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:40.381751 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637\": container with ID starting with 7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637 not found: ID does not exist" containerID="7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637" Oct 30 21:59:40.381815 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:40.381780 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637"} err="failed to get container status \"7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637\": rpc error: code = NotFound desc = could not find container \"7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637\": container with ID starting with 7da2dceb3c793f5879c085258f776d5be381fb753ca0fad66a6a4195dcbd5637 not found: ID does not exist" Oct 30 21:59:42.344894 ip-10-0-142-119 kubenswrapper[5914]: E1030 21:59:42.344844 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 21:59:52.581697 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:52.581661 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 21:59:56.078187 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078151 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s"] Oct 30 21:59:56.078704 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078690 5914 cpu_manager.go:414] "containerMap: removing container" podUID="457f7406-0588-4389-9731-550c7ae5f988" containerName="payload-pre-processing" Oct 30 21:59:56.078775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078706 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457f7406-0588-4389-9731-550c7ae5f988" containerName="payload-pre-processing" Oct 30 21:59:56.078775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078720 5914 cpu_manager.go:414] "containerMap: removing container" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" containerName="test-curl-7301-73701a" Oct 30 21:59:56.078775 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078725 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" containerName="test-curl-7301-73701a" Oct 30 21:59:56.078924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078800 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="457f7406-0588-4389-9731-550c7ae5f988" containerName="payload-pre-processing" Oct 30 21:59:56.078924 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.078815 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="377e5e8e-879f-422f-b112-8393cae3a2a7" containerName="test-curl-7301-73701a" Oct 30 21:59:56.082163 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.082145 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.165646 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.165611 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s"] Oct 30 21:59:56.207286 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207258 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207301 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207491 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207448 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207560 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207596 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207587 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207608 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrq6\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207707 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207633 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207738 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.207777 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.207764 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309260 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309218 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309280 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309454 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309309 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrq6\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309456 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309536 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309587 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309574 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309631 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309742 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309682 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309726 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309748 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.309852 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309786 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.310305 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.309862 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.319769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.310626 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.319769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.311031 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.319769 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.313677 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.321010 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.320984 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.373402 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.373365 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrq6\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.374058 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.374033 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token\") pod \"e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.396056 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.396032 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:56.826694 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.826659 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s"] Oct 30 21:59:56.828433 ip-10-0-142-119 kubenswrapper[5914]: W1030 21:59:56.828404 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277f14e0_ae56_4e24_88d6_fcb6751e9dd7.slice/crio-2f44c035dade6d2fbe25c89933b3a3e4e3795862b7ddff7fa17a67e70ef9b477 WatchSource:0}: Error finding container 2f44c035dade6d2fbe25c89933b3a3e4e3795862b7ddff7fa17a67e70ef9b477: Status 404 returned error can't find the container with id 2f44c035dade6d2fbe25c89933b3a3e4e3795862b7ddff7fa17a67e70ef9b477 Oct 30 21:59:56.830782 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.830746 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:56.830910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.830816 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:56.830910 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:56.830843 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:57.390818 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.390786 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq"] Oct 30 21:59:57.391307 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.390998 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" containerID="cri-o://8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452" gracePeriod=30 Oct 30 21:59:57.419632 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.419601 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s"] Oct 30 21:59:57.472129 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.472082 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" event={"ID":"277f14e0-ae56-4e24-88d6-fcb6751e9dd7","Type":"ContainerStarted","Data":"72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39"} Oct 30 21:59:57.472311 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.472163 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" event={"ID":"277f14e0-ae56-4e24-88d6-fcb6751e9dd7","Type":"ContainerStarted","Data":"2f44c035dade6d2fbe25c89933b3a3e4e3795862b7ddff7fa17a67e70ef9b477"} Oct 30 21:59:57.472358 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.472334 5914 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" secret="" err="secret \"e2e-worker-w0-1b5b5f-openshift-default-dockercfg-kzsbh\" not found" Oct 30 21:59:57.639068 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:57.639012 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" podStartSLOduration=1.638996012 podStartE2EDuration="1.638996012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:57.637257147 +0000 UTC m=+2549.539121213" watchObservedRunningTime="2026-10-30 21:59:57.638996012 +0000 UTC m=+2549.540860077" Oct 30 21:59:58.396756 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:58.396721 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:58.402106 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:58.402078 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:58.477991 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:58.477919 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" podUID="277f14e0-ae56-4e24-88d6-fcb6751e9dd7" containerName="istio-proxy" containerID="cri-o://72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39" gracePeriod=30 Oct 30 21:59:58.478204 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:58.478002 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 21:59:58.479110 ip-10-0-142-119 kubenswrapper[5914]: I1030 21:59:58.479088 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 22:00:02.663530 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.663464 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 22:00:02.677739 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677705 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.677957 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677779 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.677957 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677803 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.677957 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677822 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.677957 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677856 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.678189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677960 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.678189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.677986 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.678189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678027 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.678189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678085 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:02.678189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678111 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data\") pod \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\" (UID: \"37547fb6-b031-4e65-ab9a-4bfa41ea3056\") " Oct 30 22:00:02.678542 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678296 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:02.678542 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678524 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:02.678656 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678552 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.678656 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678574 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.678656 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678579 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:02.678656 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.678623 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:02.680391 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.680361 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:02.680524 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.680402 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:02.680524 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.680461 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:02.680524 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.680493 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt" pod "37547fb6-b031-4e65-ab9a-4bfa41ea3056" (UID: "37547fb6-b031-4e65-ab9a-4bfa41ea3056"). InnerVolumeSpecName "kube-api-access-zj2zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:02.779983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.779945 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.779983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.779977 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.779983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.779988 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.780230 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.779998 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.780230 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.780008 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37547fb6-b031-4e65-ab9a-4bfa41ea3056-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.780230 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.780018 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2zt\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-kube-api-access-zj2zt\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:02.780230 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:02.780028 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37547fb6-b031-4e65-ab9a-4bfa41ea3056-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.506910 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.506863 5914 generic.go:356] "Generic (PLEG): container finished" podID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerID="8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452" exitCode=0 Oct 30 22:00:03.507083 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.506944 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" Oct 30 22:00:03.507083 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.506945 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" event={"ID":"37547fb6-b031-4e65-ab9a-4bfa41ea3056","Type":"ContainerDied","Data":"8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452"} Oct 30 22:00:03.507083 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.507009 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq" event={"ID":"37547fb6-b031-4e65-ab9a-4bfa41ea3056","Type":"ContainerDied","Data":"ecdb81f940bf936759d0b45f5beb06015c6f5fb4c13f01fddeae4695d0f78af6"} Oct 30 22:00:03.507083 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.507045 5914 scope.go:122] "RemoveContainer" containerID="8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452" Oct 30 22:00:03.590533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.590378 5914 scope.go:122] "RemoveContainer" containerID="107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986" Oct 30 22:00:03.600264 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.600241 5914 scope.go:122] "RemoveContainer" containerID="8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452" Oct 30 22:00:03.600576 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:03.600553 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452\": container with ID starting with 8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452 not found: ID does not exist" containerID="8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452" Oct 30 22:00:03.600660 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.600590 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452"} err="failed to get container status \"8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452\": rpc error: code = NotFound desc = could not find container \"8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452\": container with ID starting with 8c649171e034ff49d0f6cde343a2de03cb320be96607155f24f2fb2910e82452 not found: ID does not exist" Oct 30 22:00:03.600660 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.600615 5914 scope.go:122] "RemoveContainer" containerID="107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986" Oct 30 22:00:03.600931 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:03.600875 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986\": container with ID starting with 107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986 not found: ID does not exist" containerID="107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986" Oct 30 22:00:03.600988 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.600941 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986"} err="failed to get container status \"107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986\": rpc error: code = NotFound desc = could not find container \"107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986\": container with ID starting with 107ca7cad5ae996f0047eac4e2e53727ee4c436d7fc954dfe1f60a45ee15a986 not found: ID does not exist" Oct 30 22:00:03.736048 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.735979 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq"] Oct 30 22:00:03.742467 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.742440 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 22:00:03.743052 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.743031 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-vvnjq"] Oct 30 22:00:03.791533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791494 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791531 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791783 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791553 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791783 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791586 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791783 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791639 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791783 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791783 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791701 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.791783 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791745 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.792143 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791826 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data\") pod \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\" (UID: \"277f14e0-ae56-4e24-88d6-fcb6751e9dd7\") " Oct 30 22:00:03.792143 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791823 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.792143 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791853 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.792143 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.791924 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792166 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792171 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792270 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792281 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792294 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792304 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.792356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.792313 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.794278 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.794247 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:03.794393 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.794287 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "kube-api-access-ldrq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:03.794545 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.794523 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:03.794588 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.794530 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo" pod "277f14e0-ae56-4e24-88d6-fcb6751e9dd7" (UID: "277f14e0-ae56-4e24-88d6-fcb6751e9dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:03.893417 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.893377 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.893417 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.893415 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.893622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.893427 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrq6\" (UniqueName: \"kubernetes.io/projected/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-kube-api-access-ldrq6\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:03.893622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:03.893439 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277f14e0-ae56-4e24-88d6-fcb6751e9dd7-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:04.127985 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.127853 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7"] Oct 30 22:00:04.128563 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128541 5914 cpu_manager.go:414] "containerMap: removing container" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" Oct 30 22:00:04.128647 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128568 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" Oct 30 22:00:04.128689 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128649 5914 cpu_manager.go:414] "containerMap: removing container" podUID="277f14e0-ae56-4e24-88d6-fcb6751e9dd7" containerName="istio-proxy" Oct 30 22:00:04.128689 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128657 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277f14e0-ae56-4e24-88d6-fcb6751e9dd7" containerName="istio-proxy" Oct 30 22:00:04.128770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128749 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" Oct 30 22:00:04.128770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128758 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="277f14e0-ae56-4e24-88d6-fcb6751e9dd7" containerName="istio-proxy" Oct 30 22:00:04.128770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.128764 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" Oct 30 22:00:04.132497 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.132471 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.196375 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.196107 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7"] Oct 30 22:00:04.196375 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.196325 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6t2c7\"" Oct 30 22:00:04.196375 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.196377 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:00:04.196975 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.196953 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.198000 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.197015 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr7vf\" (UniqueName: \"kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.198000 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.197101 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.218031 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.217997 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" path="/var/lib/kubelet/pods/37547fb6-b031-4e65-ab9a-4bfa41ea3056/volumes" Oct 30 22:00:04.298548 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.298505 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr7vf\" (UniqueName: \"kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.298734 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.298577 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.298837 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.298753 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.299377 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.299351 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.301130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.301110 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.358726 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.358700 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr7vf\" (UniqueName: \"kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf\") pod \"maas-api-revoke-sub-15b2714a-j6bz7\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.446942 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.446859 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:04.518683 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.518655 5914 generic.go:356] "Generic (PLEG): container finished" podID="277f14e0-ae56-4e24-88d6-fcb6751e9dd7" containerID="72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39" exitCode=0 Oct 30 22:00:04.518842 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.518831 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" Oct 30 22:00:04.522127 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.519449 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" event={"ID":"277f14e0-ae56-4e24-88d6-fcb6751e9dd7","Type":"ContainerDied","Data":"72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39"} Oct 30 22:00:04.522127 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.519518 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s" event={"ID":"277f14e0-ae56-4e24-88d6-fcb6751e9dd7","Type":"ContainerDied","Data":"2f44c035dade6d2fbe25c89933b3a3e4e3795862b7ddff7fa17a67e70ef9b477"} Oct 30 22:00:04.522127 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.519548 5914 scope.go:122] "RemoveContainer" containerID="72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39" Oct 30 22:00:04.533297 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.533269 5914 scope.go:122] "RemoveContainer" containerID="72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39" Oct 30 22:00:04.533661 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:04.533636 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39\": container with ID starting with 72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39 not found: ID does not exist" containerID="72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39" Oct 30 22:00:04.533744 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.533669 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39"} err="failed to get container status \"72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39\": rpc error: code = NotFound desc = could not find container \"72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39\": container with ID starting with 72ddd4c3f11da465bc1f2d1d69c20f3afcec0b45c019b8bb8ee51311180a3f39 not found: ID does not exist" Oct 30 22:00:04.690090 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.690050 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s"] Oct 30 22:00:04.700332 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.700255 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1b5b5f-openshift-default-59bd77464c-wnp4s"] Oct 30 22:00:04.842394 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:04.842371 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7"] Oct 30 22:00:04.845334 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:00:04.845303 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c12d75c_5433_4ade_b30b_d5c114ff04ee.slice/crio-1b3e4e9decfd2d8097705ecee90c814369b9cb1ab6732bca260e04691c27c61c WatchSource:0}: Error finding container 1b3e4e9decfd2d8097705ecee90c814369b9cb1ab6732bca260e04691c27c61c: Status 404 returned error can't find the container with id 1b3e4e9decfd2d8097705ecee90c814369b9cb1ab6732bca260e04691c27c61c Oct 30 22:00:05.530825 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:05.530760 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" event={"ID":"7c12d75c-5433-4ade-b30b-d5c114ff04ee","Type":"ContainerStarted","Data":"4b57f9ba738f4c8b1b0d23c5123fabfb83fe68278a2f048a82183f9bbbeac15c"} Oct 30 22:00:05.531043 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:05.530871 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" event={"ID":"7c12d75c-5433-4ade-b30b-d5c114ff04ee","Type":"ContainerStarted","Data":"1b3e4e9decfd2d8097705ecee90c814369b9cb1ab6732bca260e04691c27c61c"} Oct 30 22:00:05.663959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:05.663907 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" podStartSLOduration=1.663870949 podStartE2EDuration="1.663870949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:05.662019892 +0000 UTC m=+2557.563883958" watchObservedRunningTime="2026-10-30 22:00:05.663870949 +0000 UTC m=+2557.565735013" Oct 30 22:00:06.217443 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:06.217391 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277f14e0-ae56-4e24-88d6-fcb6751e9dd7" path="/var/lib/kubelet/pods/277f14e0-ae56-4e24-88d6-fcb6751e9dd7/volumes" Oct 30 22:00:06.537948 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:06.537836 5914 generic.go:356] "Generic (PLEG): container finished" podID="7c12d75c-5433-4ade-b30b-d5c114ff04ee" containerID="4b57f9ba738f4c8b1b0d23c5123fabfb83fe68278a2f048a82183f9bbbeac15c" exitCode=0 Oct 30 22:00:06.537948 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:06.537926 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" event={"ID":"7c12d75c-5433-4ade-b30b-d5c114ff04ee","Type":"ContainerDied","Data":"4b57f9ba738f4c8b1b0d23c5123fabfb83fe68278a2f048a82183f9bbbeac15c"} Oct 30 22:00:07.685213 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.685185 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:07.734905 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.734863 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf\" (UniqueName: \"kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf\") pod \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " Oct 30 22:00:07.735049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.734933 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca\") pod \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " Oct 30 22:00:07.735049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.734961 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca\") pod \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\" (UID: \"7c12d75c-5433-4ade-b30b-d5c114ff04ee\") " Oct 30 22:00:07.735329 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.735297 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca" pod "7c12d75c-5433-4ade-b30b-d5c114ff04ee" (UID: "7c12d75c-5433-4ade-b30b-d5c114ff04ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:07.735475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.735458 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-openshift-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.737228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.737170 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf" pod "7c12d75c-5433-4ade-b30b-d5c114ff04ee" (UID: "7c12d75c-5433-4ade-b30b-d5c114ff04ee"). InnerVolumeSpecName "kube-api-access-cr7vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:07.737228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.737223 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca" pod "7c12d75c-5433-4ade-b30b-d5c114ff04ee" (UID: "7c12d75c-5433-4ade-b30b-d5c114ff04ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:07.836858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.836782 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr7vf\" (UniqueName: \"kubernetes.io/projected/7c12d75c-5433-4ade-b30b-d5c114ff04ee-kube-api-access-cr7vf\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.836858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.836810 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c12d75c-5433-4ade-b30b-d5c114ff04ee-maas-api-xks-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:07.857893 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.857850 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg"] Oct 30 22:00:07.858073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:07.858053 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" podUID="737311ce-3c3a-4cd9-a954-a889768bdb67" containerName="discovery" containerID="cri-o://a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7" gracePeriod=30 Oct 30 22:00:08.098619 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.098593 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 22:00:08.140138 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140100 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140162 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140205 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140233 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140306 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140543 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140464 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140543 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140529 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts\") pod \"737311ce-3c3a-4cd9-a954-a889768bdb67\" (UID: \"737311ce-3c3a-4cd9-a954-a889768bdb67\") " Oct 30 22:00:08.140724 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.140702 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:08.141030 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.141014 5914 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-ca-configmap\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.142655 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.142627 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:08.143049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.143016 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:08.143049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.143037 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:08.143202 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.143085 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:08.143202 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.143178 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:08.143202 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.143198 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv" pod "737311ce-3c3a-4cd9-a954-a889768bdb67" (UID: "737311ce-3c3a-4cd9-a954-a889768bdb67"). InnerVolumeSpecName "kube-api-access-d6hlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:08.242073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.241941 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.242073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.241975 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6hlv\" (UniqueName: \"kubernetes.io/projected/737311ce-3c3a-4cd9-a954-a889768bdb67-kube-api-access-d6hlv\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.242073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.241991 5914 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-kubeconfig\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.242073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.242004 5914 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-istio-csr-dns-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.242073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.242019 5914 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/737311ce-3c3a-4cd9-a954-a889768bdb67-local-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.242073 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.242032 5914 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/737311ce-3c3a-4cd9-a954-a889768bdb67-cacerts\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.550080 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.550047 5914 generic.go:356] "Generic (PLEG): container finished" podID="737311ce-3c3a-4cd9-a954-a889768bdb67" containerID="a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7" exitCode=0 Oct 30 22:00:08.550305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.550096 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" Oct 30 22:00:08.550305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.550138 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" event={"ID":"737311ce-3c3a-4cd9-a954-a889768bdb67","Type":"ContainerDied","Data":"a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7"} Oct 30 22:00:08.550305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.550197 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg" event={"ID":"737311ce-3c3a-4cd9-a954-a889768bdb67","Type":"ContainerDied","Data":"69e345c1ab9d01b07dd7bf6e0bc29e05aa2f85d5ab25576606d457ad2252e375"} Oct 30 22:00:08.550305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.550224 5914 scope.go:122] "RemoveContainer" containerID="a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7" Oct 30 22:00:08.551770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.551745 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" event={"ID":"7c12d75c-5433-4ade-b30b-d5c114ff04ee","Type":"ContainerDied","Data":"1b3e4e9decfd2d8097705ecee90c814369b9cb1ab6732bca260e04691c27c61c"} Oct 30 22:00:08.551931 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.551780 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7" Oct 30 22:00:08.551931 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.551791 5914 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b3e4e9decfd2d8097705ecee90c814369b9cb1ab6732bca260e04691c27c61c" Oct 30 22:00:08.566753 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.566734 5914 scope.go:122] "RemoveContainer" containerID="a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7" Oct 30 22:00:08.567039 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:08.567020 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7\": container with ID starting with a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7 not found: ID does not exist" containerID="a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7" Oct 30 22:00:08.567094 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.567048 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7"} err="failed to get container status \"a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7\": rpc error: code = NotFound desc = could not find container \"a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7\": container with ID starting with a686036ade526f09c2400b559610899a7195908a0f5281bab0630ff29c2c1fc7 not found: ID does not exist" Oct 30 22:00:08.721632 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.721599 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg"] Oct 30 22:00:08.727386 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:08.727360 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-v6qfg"] Oct 30 22:00:10.223593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:10.223550 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="737311ce-3c3a-4cd9-a954-a889768bdb67" path="/var/lib/kubelet/pods/737311ce-3c3a-4cd9-a954-a889768bdb67/volumes" Oct 30 22:00:12.170759 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:12.170722 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:00:12.345073 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:12.345032 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:00:32.012132 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:32.012094 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:00:34.246475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.246441 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx"] Oct 30 22:00:34.246948 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.246661 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" podUID="447cbf22-f34e-4c95-ac87-127d1f9ad06c" containerName="maas-api" containerID="cri-o://ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c" gracePeriod=30 Oct 30 22:00:34.518312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.518286 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 22:00:34.524357 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.524330 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs\") pod \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " Oct 30 22:00:34.524493 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.524412 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9\" (UniqueName: \"kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9\") pod \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " Oct 30 22:00:34.524493 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.524477 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\") pod \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " Oct 30 22:00:34.524572 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.524509 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca\") pod \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\" (UID: \"447cbf22-f34e-4c95-ac87-127d1f9ad06c\") " Oct 30 22:00:34.525012 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.524946 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca" pod "447cbf22-f34e-4c95-ac87-127d1f9ad06c" (UID: "447cbf22-f34e-4c95-ac87-127d1f9ad06c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:34.526681 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.526655 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9" pod "447cbf22-f34e-4c95-ac87-127d1f9ad06c" (UID: "447cbf22-f34e-4c95-ac87-127d1f9ad06c"). InnerVolumeSpecName "kube-api-access-f5jd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:34.526681 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.526661 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs" pod "447cbf22-f34e-4c95-ac87-127d1f9ad06c" (UID: "447cbf22-f34e-4c95-ac87-127d1f9ad06c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:34.526838 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.526752 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls" pod "447cbf22-f34e-4c95-ac87-127d1f9ad06c" (UID: "447cbf22-f34e-4c95-ac87-127d1f9ad06c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:00:34.625654 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.625614 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:34.625654 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.625647 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/447cbf22-f34e-4c95-ac87-127d1f9ad06c-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:34.625654 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.625658 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/447cbf22-f34e-4c95-ac87-127d1f9ad06c-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:34.625942 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.625668 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jd9\" (UniqueName: \"kubernetes.io/projected/447cbf22-f34e-4c95-ac87-127d1f9ad06c-kube-api-access-f5jd9\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:34.690294 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.690262 5914 generic.go:356] "Generic (PLEG): container finished" podID="447cbf22-f34e-4c95-ac87-127d1f9ad06c" containerID="ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c" exitCode=0 Oct 30 22:00:34.690523 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.690315 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" Oct 30 22:00:34.690523 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.690338 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" event={"ID":"447cbf22-f34e-4c95-ac87-127d1f9ad06c","Type":"ContainerDied","Data":"ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c"} Oct 30 22:00:34.690523 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.690392 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx" event={"ID":"447cbf22-f34e-4c95-ac87-127d1f9ad06c","Type":"ContainerDied","Data":"efb4d1143a30666fe6578c9625e09a668512f26c6f822121fa3a979c318045b1"} Oct 30 22:00:34.690523 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.690414 5914 scope.go:122] "RemoveContainer" containerID="ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c" Oct 30 22:00:34.700659 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.700640 5914 scope.go:122] "RemoveContainer" containerID="ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c" Oct 30 22:00:34.700945 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:34.700924 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c\": container with ID starting with ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c not found: ID does not exist" containerID="ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c" Oct 30 22:00:34.701023 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.700959 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c"} err="failed to get container status \"ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c\": rpc error: code = NotFound desc = could not find container \"ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c\": container with ID starting with ba38caac31eca48f909c101491258d55acaeb3b49a2f8c753b1a9c729642387c not found: ID does not exist" Oct 30 22:00:34.872171 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.872090 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx"] Oct 30 22:00:34.878137 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:34.878108 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-fsqkx"] Oct 30 22:00:36.216318 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:00:36.216279 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447cbf22-f34e-4c95-ac87-127d1f9ad06c" path="/var/lib/kubelet/pods/447cbf22-f34e-4c95-ac87-127d1f9ad06c/volumes" Oct 30 22:00:42.345860 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:00:42.345818 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:01:00.204898 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:00.204853 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:01:07.568683 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.568598 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh"] Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569124 5914 cpu_manager.go:414] "containerMap: removing container" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569140 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37547fb6-b031-4e65-ab9a-4bfa41ea3056" containerName="istio-proxy" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569152 5914 cpu_manager.go:414] "containerMap: removing container" podUID="737311ce-3c3a-4cd9-a954-a889768bdb67" containerName="discovery" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569157 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="737311ce-3c3a-4cd9-a954-a889768bdb67" containerName="discovery" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569168 5914 cpu_manager.go:414] "containerMap: removing container" podUID="7c12d75c-5433-4ade-b30b-d5c114ff04ee" containerName="revoke-keys" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569173 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c12d75c-5433-4ade-b30b-d5c114ff04ee" containerName="revoke-keys" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569186 5914 cpu_manager.go:414] "containerMap: removing container" podUID="447cbf22-f34e-4c95-ac87-127d1f9ad06c" containerName="maas-api" Oct 30 22:01:07.569211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569190 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447cbf22-f34e-4c95-ac87-127d1f9ad06c" containerName="maas-api" Oct 30 22:01:07.569474 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569271 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="737311ce-3c3a-4cd9-a954-a889768bdb67" containerName="discovery" Oct 30 22:01:07.569474 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569285 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c12d75c-5433-4ade-b30b-d5c114ff04ee" containerName="revoke-keys" Oct 30 22:01:07.569474 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.569297 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="447cbf22-f34e-4c95-ac87-127d1f9ad06c" containerName="maas-api" Oct 30 22:01:07.572748 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.572726 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.678289 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.678250 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-75c2acf6\"" Oct 30 22:01:07.678621 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.678604 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-75c2acf6-dockercfg-ldxsp\"" Oct 30 22:01:07.680121 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.680091 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh"] Oct 30 22:01:07.689751 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.689719 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h"] Oct 30 22:01:07.693511 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.693491 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:07.757257 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.757221 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h"] Oct 30 22:01:07.765047 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.765016 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.765208 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.765067 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdv7\" (UniqueName: \"kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.865808 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.865724 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.865808 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.865770 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpsv\" (UniqueName: \"kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:07.866058 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.865815 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdv7\" (UniqueName: \"kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.866058 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.866029 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:07.866787 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.866612 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.934516 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.934480 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdv7\" (UniqueName: \"kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7\") pod \"payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:07.966769 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.966733 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpsv\" (UniqueName: \"kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:07.966949 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.966845 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:07.967497 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:07.967478 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:08.026398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.026352 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpsv\" (UniqueName: \"kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv\") pod \"payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:08.186074 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.186038 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:08.306410 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.306364 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:08.579543 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.579514 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh"] Oct 30 22:01:08.580416 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:01:08.580387 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9549c05c_a3e7_4349_a48f_838c2617f602.slice/crio-82cbd13fd5486008db0f7cb4f281853dc63d995dce3d08f1abd579015e9ca2b0 WatchSource:0}: Error finding container 82cbd13fd5486008db0f7cb4f281853dc63d995dce3d08f1abd579015e9ca2b0: Status 404 returned error can't find the container with id 82cbd13fd5486008db0f7cb4f281853dc63d995dce3d08f1abd579015e9ca2b0 Oct 30 22:01:08.702788 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.702757 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h"] Oct 30 22:01:08.703752 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:01:08.703728 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749695c2_24eb_4ecd_b8cb_870239a8b36a.slice/crio-e3058b1ed686a8a6a4831654c91493c09cf498b24f5e3bdae2686e8d051025f0 WatchSource:0}: Error finding container e3058b1ed686a8a6a4831654c91493c09cf498b24f5e3bdae2686e8d051025f0: Status 404 returned error can't find the container with id e3058b1ed686a8a6a4831654c91493c09cf498b24f5e3bdae2686e8d051025f0 Oct 30 22:01:08.877517 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.877476 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" event={"ID":"749695c2-24eb-4ecd-b8cb-870239a8b36a","Type":"ContainerStarted","Data":"55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2"} Oct 30 22:01:08.877713 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.877530 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" event={"ID":"749695c2-24eb-4ecd-b8cb-870239a8b36a","Type":"ContainerStarted","Data":"e3058b1ed686a8a6a4831654c91493c09cf498b24f5e3bdae2686e8d051025f0"} Oct 30 22:01:08.877713 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.877597 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:08.879297 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.879271 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" event={"ID":"9549c05c-a3e7-4349-a48f-838c2617f602","Type":"ContainerStarted","Data":"2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755"} Oct 30 22:01:08.879416 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.879317 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:08.879416 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:08.879334 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" event={"ID":"9549c05c-a3e7-4349-a48f-838c2617f602","Type":"ContainerStarted","Data":"82cbd13fd5486008db0f7cb4f281853dc63d995dce3d08f1abd579015e9ca2b0"} Oct 30 22:01:09.136963 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:09.136849 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" podStartSLOduration=2.136828974 podStartE2EDuration="2.136828974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:09.135517226 +0000 UTC m=+2621.037381291" watchObservedRunningTime="2026-10-30 22:01:09.136828974 +0000 UTC m=+2621.038693070" Oct 30 22:01:09.136963 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:09.136951 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" podStartSLOduration=2.136946966 podStartE2EDuration="2.136946966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:09.003410386 +0000 UTC m=+2620.905274452" watchObservedRunningTime="2026-10-30 22:01:09.136946966 +0000 UTC m=+2621.038811031" Oct 30 22:01:12.345559 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:01:12.345516 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:01:19.887777 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:19.887746 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:01:19.888260 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:19.887814 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:01:21.260134 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:21.260097 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:01:30.818836 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.818806 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn"] Oct 30 22:01:30.823634 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.823618 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:30.878559 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.878526 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-75c2acf6-serving-cert\"" Oct 30 22:01:30.881143 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.881112 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn"] Oct 30 22:01:30.903864 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.903827 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:30.903864 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.903862 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:30.904091 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.903917 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:30.904130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:30.904091 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xd2\" (UniqueName: \"kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.005679 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.005640 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.005679 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.005678 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.006018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.005708 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.006018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.005753 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xd2\" (UniqueName: \"kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.006018 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:01:31.005790 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-serving-cert: secret "maas-api-e2e-mt-75c2acf6-serving-cert" not found Oct 30 22:01:31.006018 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:01:31.005860 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls podName:f1715934-01e0-41b2-a2d2-a16265815f8d nodeName:}" failed. No retries permitted until 2026-10-30 22:01:31.505843284 +0000 UTC m=+2643.407707339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls") pod "maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" (UID: "f1715934-01e0-41b2-a2d2-a16265815f8d") : secret "maas-api-e2e-mt-75c2acf6-serving-cert" not found Oct 30 22:01:31.006441 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.006415 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.008358 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.008342 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.063263 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.063225 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xd2\" (UniqueName: \"kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.510207 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.510176 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.512717 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.512688 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\") pod \"maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:31.736987 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:31.736951 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:32.109072 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:32.109019 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn"] Oct 30 22:01:32.111601 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:01:32.111567 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1715934_01e0_41b2_a2d2_a16265815f8d.slice/crio-4c8822ba4fe19d54b682a09007c3d31cba1b886974336c01f9545fbc6d17060e WatchSource:0}: Error finding container 4c8822ba4fe19d54b682a09007c3d31cba1b886974336c01f9545fbc6d17060e: Status 404 returned error can't find the container with id 4c8822ba4fe19d54b682a09007c3d31cba1b886974336c01f9545fbc6d17060e Oct 30 22:01:33.003627 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:33.003593 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" event={"ID":"f1715934-01e0-41b2-a2d2-a16265815f8d","Type":"ContainerStarted","Data":"4c8822ba4fe19d54b682a09007c3d31cba1b886974336c01f9545fbc6d17060e"} Oct 30 22:01:34.010378 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:34.010335 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" event={"ID":"f1715934-01e0-41b2-a2d2-a16265815f8d","Type":"ContainerStarted","Data":"fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad"} Oct 30 22:01:34.010772 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:34.010488 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:34.127513 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:34.127456 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" podStartSLOduration=3.2769752 podStartE2EDuration="4.127440758s" totalImagesPullingTime="850.465558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:34.126303022 +0000 UTC m=+2646.028167088" watchObservedRunningTime="2026-10-30 22:01:34.127440758 +0000 UTC m=+2646.029304823" Oct 30 22:01:40.021036 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:40.021008 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:01:40.952840 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:40.952804 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:01:42.346337 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:01:42.346297 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:01:54.259386 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.259352 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z"] Oct 30 22:01:54.263160 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.263143 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.309157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.309123 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd"] Oct 30 22:01:54.313063 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.313048 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.328126 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.328105 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-75c2acf6-openshift-default-dockercfg-q8244\"" Oct 30 22:01:54.337639 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337612 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.337750 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337670 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.337750 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337689 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.337839 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337799 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.337839 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337830 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.337942 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337852 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.337988 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337944 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.337988 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.337975 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338001 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.338065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338018 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338057 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.338175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338071 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338096 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338133 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthrw\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.338286 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338191 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338286 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338272 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2xd\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338299 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.338363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.338329 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.365419 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.365375 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z"] Oct 30 22:01:54.413276 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.413235 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd"] Oct 30 22:01:54.439353 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439316 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.439538 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439371 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.439538 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439415 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439538 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439451 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.439538 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439479 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439538 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439516 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439541 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439579 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439605 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthrw\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439632 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439680 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2xd\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439722 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439779 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.439817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439804 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.440299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439847 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.440299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439893 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.440299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.439916 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.440299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440141 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.440299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440287 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.440577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440324 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.440577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440468 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.440577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440558 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.440577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440568 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.440843 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440816 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.440961 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.440939 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.441142 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.441085 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.441260 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.441198 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.441520 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.441494 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.442525 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.442502 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.442742 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.442717 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.442935 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.442873 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.443084 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.443066 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.497100 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.497068 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.497302 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.497282 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.540232 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.540168 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthrw\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:54.540370 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.540241 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2xd\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.577373 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.577336 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:54.625701 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:54.625664 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:55.004314 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.004289 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z"] Oct 30 22:01:55.006278 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:01:55.006251 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b36b4f0_2032_45ef_9d3e_50e269b84dd0.slice/crio-f722b4634b23a7c57cb2cb3fa76603010a2b9298866eb00c6b39bfab3947be8e WatchSource:0}: Error finding container f722b4634b23a7c57cb2cb3fa76603010a2b9298866eb00c6b39bfab3947be8e: Status 404 returned error can't find the container with id f722b4634b23a7c57cb2cb3fa76603010a2b9298866eb00c6b39bfab3947be8e Oct 30 22:01:55.008617 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.008585 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:01:55.008708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.008650 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:01:55.008708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.008681 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:01:55.059325 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.059142 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd"] Oct 30 22:01:55.061991 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:01:55.061946 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21043b4_66b0_4258_8cda_ee9e95f8ca59.slice/crio-9b797114ad210a43ac4b76ca0712390b3a41ef7218f66d8a672728fa83ee634e WatchSource:0}: Error finding container 9b797114ad210a43ac4b76ca0712390b3a41ef7218f66d8a672728fa83ee634e: Status 404 returned error can't find the container with id 9b797114ad210a43ac4b76ca0712390b3a41ef7218f66d8a672728fa83ee634e Oct 30 22:01:55.065344 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.065291 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:01:55.065470 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.065442 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:01:55.065542 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.065524 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:01:55.120785 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.120744 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" event={"ID":"b21043b4-66b0-4258-8cda-ee9e95f8ca59","Type":"ContainerStarted","Data":"9b797114ad210a43ac4b76ca0712390b3a41ef7218f66d8a672728fa83ee634e"} Oct 30 22:01:55.123033 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.122973 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" event={"ID":"8b36b4f0-2032-45ef-9d3e-50e269b84dd0","Type":"ContainerStarted","Data":"f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64"} Oct 30 22:01:55.123033 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.123031 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" event={"ID":"8b36b4f0-2032-45ef-9d3e-50e269b84dd0","Type":"ContainerStarted","Data":"f722b4634b23a7c57cb2cb3fa76603010a2b9298866eb00c6b39bfab3947be8e"} Oct 30 22:01:55.241397 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.241312 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podStartSLOduration=1.241290365 podStartE2EDuration="1.241290365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:55.240147846 +0000 UTC m=+2667.142011913" watchObservedRunningTime="2026-10-30 22:01:55.241290365 +0000 UTC m=+2667.143154432" Oct 30 22:01:55.589907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:55.585011 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:01:56.131423 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.131377 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" event={"ID":"b21043b4-66b0-4258-8cda-ee9e95f8ca59","Type":"ContainerStarted","Data":"51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092"} Oct 30 22:01:56.253939 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.253854 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podStartSLOduration=2.253833566 podStartE2EDuration="2.253833566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:56.252022931 +0000 UTC m=+2668.153886997" watchObservedRunningTime="2026-10-30 22:01:56.253833566 +0000 UTC m=+2668.155697635" Oct 30 22:01:56.588631 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.588538 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:01:56.592907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.588904 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:01:56.626908 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.626780 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:01:56.628487 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.628458 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:01:56.628617 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:56.628516 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:01:57.583345 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:57.583302 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:01:57.583757 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:57.583726 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:01:57.626789 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:57.626748 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:01:57.627442 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:57.627412 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:01:58.597057 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:58.597012 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:01:58.597363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:58.597333 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:01:58.627079 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:58.627040 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:01:58.627677 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:58.627630 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:01:59.599309 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:59.599269 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:01:59.599611 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:59.599580 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:01:59.626711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:59.626679 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:01:59.626944 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:01:59.626920 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:02:00.581357 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:00.581313 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:02:00.582015 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:00.581973 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:02:00.626216 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:00.626177 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:02:00.626703 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:00.626670 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 22:02:01.976162 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:01.976124 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:02:01.977109 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:01.977084 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:02:01.994080 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:01.994044 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:02:02.008952 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:02.008920 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:02:02.009241 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:02.009221 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:02:02.010093 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:02.010070 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:02:02.568139 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:02.568105 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg"] Oct 30 22:02:08.040808 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:08.040773 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:02:09.400173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.400135 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2"] Oct 30 22:02:09.405315 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.405293 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445612 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445574 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445612 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445616 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445833 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445658 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445833 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445698 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445833 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445760 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445833 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445784 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh28\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.445833 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445811 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.446080 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445849 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.446080 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.445903 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.509732 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.509694 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2"] Oct 30 22:02:09.547335 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547294 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547343 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547382 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547432 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547502 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547539 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh28\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547576 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547616 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547661 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.547768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547757 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.548064 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.547800 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.548064 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.548040 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.548181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.548142 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.548293 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.548270 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.550312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.550286 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.550760 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.550741 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.604236 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.604198 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.604623 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.604598 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh28\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:09.719351 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:09.719252 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:10.144676 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:10.144640 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2"] Oct 30 22:02:10.186488 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:02:10.186451 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad69e28_4b1e_4aa5_911e_f7456dbc36fb.slice/crio-6484b395b5a9c520a2309cc4d0002f0e14e5aa2ef9695ee7a353f3cd3234368f WatchSource:0}: Error finding container 6484b395b5a9c520a2309cc4d0002f0e14e5aa2ef9695ee7a353f3cd3234368f: Status 404 returned error can't find the container with id 6484b395b5a9c520a2309cc4d0002f0e14e5aa2ef9695ee7a353f3cd3234368f Oct 30 22:02:10.188970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:10.188936 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.189045 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:10.189003 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.189045 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:10.189033 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:10.215976 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:10.215943 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" event={"ID":"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb","Type":"ContainerStarted","Data":"6484b395b5a9c520a2309cc4d0002f0e14e5aa2ef9695ee7a353f3cd3234368f"} Oct 30 22:02:11.219113 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:11.219075 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" event={"ID":"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb","Type":"ContainerStarted","Data":"167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab"} Oct 30 22:02:11.436598 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:11.436547 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podStartSLOduration=2.436532796 podStartE2EDuration="2.436532796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:11.385491779 +0000 UTC m=+2683.287355847" watchObservedRunningTime="2026-10-30 22:02:11.436532796 +0000 UTC m=+2683.338396861" Oct 30 22:02:11.720250 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:11.720210 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:11.722154 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:11.722120 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:02:11.722304 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:11.722168 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:02:12.345490 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:12.345450 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:02:12.720627 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:12.720594 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:02:12.720810 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:12.720647 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:02:13.720439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:13.720399 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:02:13.720851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:13.720454 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:02:14.720656 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:14.720618 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:02:14.721120 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:14.720676 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:02:15.720217 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:15.720179 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:02:15.720419 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:15.720234 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:02:16.724628 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:16.724595 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:16.725070 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:16.724897 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:16.725621 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:16.725605 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:02:17.589145 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.589095 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" podUID="d615093e-57ba-466f-9d49-1825cc8c7dee" containerName="main" containerID="cri-o://8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578" gracePeriod=45 Oct 30 22:02:17.848658 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.848596 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 22:02:17.969666 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.969628 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm\") pod \"d615093e-57ba-466f-9d49-1825cc8c7dee\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " Oct 30 22:02:17.969860 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.969682 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache\") pod \"d615093e-57ba-466f-9d49-1825cc8c7dee\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " Oct 30 22:02:17.969860 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.969719 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home\") pod \"d615093e-57ba-466f-9d49-1825cc8c7dee\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " Oct 30 22:02:17.969860 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.969788 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs\" (UniqueName: \"kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs\") pod \"d615093e-57ba-466f-9d49-1825cc8c7dee\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " Oct 30 22:02:17.969860 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.969833 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir\") pod \"d615093e-57ba-466f-9d49-1825cc8c7dee\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " Oct 30 22:02:17.970101 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.970031 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt\" (UniqueName: \"kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt\") pod \"d615093e-57ba-466f-9d49-1825cc8c7dee\" (UID: \"d615093e-57ba-466f-9d49-1825cc8c7dee\") " Oct 30 22:02:17.970672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.970150 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home" pod "d615093e-57ba-466f-9d49-1825cc8c7dee" (UID: "d615093e-57ba-466f-9d49-1825cc8c7dee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:17.970672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.970387 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir" pod "d615093e-57ba-466f-9d49-1825cc8c7dee" (UID: "d615093e-57ba-466f-9d49-1825cc8c7dee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:17.970672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.970544 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.970672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.970562 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:17.970672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.970633 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache" pod "d615093e-57ba-466f-9d49-1825cc8c7dee" (UID: "d615093e-57ba-466f-9d49-1825cc8c7dee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:17.972799 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.972768 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm" pod "d615093e-57ba-466f-9d49-1825cc8c7dee" (UID: "d615093e-57ba-466f-9d49-1825cc8c7dee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:17.972945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.972819 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt" pod "d615093e-57ba-466f-9d49-1825cc8c7dee" (UID: "d615093e-57ba-466f-9d49-1825cc8c7dee"). InnerVolumeSpecName "kube-api-access-5dbxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:17.972945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:17.972859 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs" pod "d615093e-57ba-466f-9d49-1825cc8c7dee" (UID: "d615093e-57ba-466f-9d49-1825cc8c7dee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:18.071772 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.071728 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d615093e-57ba-466f-9d49-1825cc8c7dee-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.071772 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.071765 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbxt\" (UniqueName: \"kubernetes.io/projected/d615093e-57ba-466f-9d49-1825cc8c7dee-kube-api-access-5dbxt\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.071772 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.071775 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.072068 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.071784 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d615093e-57ba-466f-9d49-1825cc8c7dee-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:18.111026 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.110938 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7301-97f2b8"] Oct 30 22:02:18.111468 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.111454 5914 cpu_manager.go:414] "containerMap: removing container" podUID="d615093e-57ba-466f-9d49-1825cc8c7dee" containerName="main" Oct 30 22:02:18.111523 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.111468 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d615093e-57ba-466f-9d49-1825cc8c7dee" containerName="main" Oct 30 22:02:18.111583 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.111574 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d615093e-57ba-466f-9d49-1825cc8c7dee" containerName="main" Oct 30 22:02:18.114871 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.114854 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:18.168155 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.168117 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7301-97f2b8"] Oct 30 22:02:18.169041 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.169009 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m2hp5\"" Oct 30 22:02:18.258122 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.258086 5914 generic.go:356] "Generic (PLEG): container finished" podID="d615093e-57ba-466f-9d49-1825cc8c7dee" containerID="8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578" exitCode=0 Oct 30 22:02:18.258337 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.258129 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" Oct 30 22:02:18.258337 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.258180 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" event={"ID":"d615093e-57ba-466f-9d49-1825cc8c7dee","Type":"ContainerDied","Data":"8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578"} Oct 30 22:02:18.258337 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.258230 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg" event={"ID":"d615093e-57ba-466f-9d49-1825cc8c7dee","Type":"ContainerDied","Data":"9f101b1afa72c5970f0331f8cddf2c7463e699609849e0da6febd15491d21b6f"} Oct 30 22:02:18.258337 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.258249 5914 scope.go:122] "RemoveContainer" containerID="8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578" Oct 30 22:02:18.267502 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.267480 5914 scope.go:122] "RemoveContainer" containerID="8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578" Oct 30 22:02:18.267822 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:18.267801 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578\": container with ID starting with 8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578 not found: ID does not exist" containerID="8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578" Oct 30 22:02:18.267894 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.267832 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578"} err="failed to get container status \"8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578\": rpc error: code = NotFound desc = could not find container \"8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578\": container with ID starting with 8c11f765808856bee500de5e5afc3b8781707947298895794d0fa87f0acfd578 not found: ID does not exist" Oct 30 22:02:18.273418 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.273385 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnwp\" (UniqueName: \"kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp\") pod \"test-curl-7301-97f2b8\" (UID: \"1f8d30be-41d8-498e-91a2-2052c69da29c\") " pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:18.375063 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.375023 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnwp\" (UniqueName: \"kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp\") pod \"test-curl-7301-97f2b8\" (UID: \"1f8d30be-41d8-498e-91a2-2052c69da29c\") " pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:18.421371 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.421335 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg"] Oct 30 22:02:18.424924 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.424893 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e9aa8e/test-model-e9aa8e-kserve-65b99bb8cc-44wwg"] Oct 30 22:02:18.432079 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.432046 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnwp\" (UniqueName: \"kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp\") pod \"test-curl-7301-97f2b8\" (UID: \"1f8d30be-41d8-498e-91a2-2052c69da29c\") " pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:18.727669 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:18.727582 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:19.104394 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:19.104369 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7301-97f2b8"] Oct 30 22:02:19.106709 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:02:19.106675 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f8d30be_41d8_498e_91a2_2052c69da29c.slice/crio-377f5635c6c2647aeeeeab7b1ea9eb207b1443d910d1f9b8cda582db22ca23fa WatchSource:0}: Error finding container 377f5635c6c2647aeeeeab7b1ea9eb207b1443d910d1f9b8cda582db22ca23fa: Status 404 returned error can't find the container with id 377f5635c6c2647aeeeeab7b1ea9eb207b1443d910d1f9b8cda582db22ca23fa Oct 30 22:02:19.266219 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:19.266179 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7301-97f2b8" event={"ID":"1f8d30be-41d8-498e-91a2-2052c69da29c","Type":"ContainerStarted","Data":"b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677"} Oct 30 22:02:19.266219 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:19.266225 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7301-97f2b8" event={"ID":"1f8d30be-41d8-498e-91a2-2052c69da29c","Type":"ContainerStarted","Data":"377f5635c6c2647aeeeeab7b1ea9eb207b1443d910d1f9b8cda582db22ca23fa"} Oct 30 22:02:19.377435 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:19.377376 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7301-97f2b8" podStartSLOduration=1.377357235 podStartE2EDuration="1.377357235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:19.377333718 +0000 UTC m=+2691.279197785" watchObservedRunningTime="2026-10-30 22:02:19.377357235 +0000 UTC m=+2691.279221302" Oct 30 22:02:20.216181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:20.216144 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d615093e-57ba-466f-9d49-1825cc8c7dee" path="/var/lib/kubelet/pods/d615093e-57ba-466f-9d49-1825cc8c7dee/volumes" Oct 30 22:02:20.807024 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:20.806988 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7301-97f2b8"] Oct 30 22:02:20.809548 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:20.809514 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7301-97f2b8"] Oct 30 22:02:21.277677 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:21.277634 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7301-97f2b8" podUID="1f8d30be-41d8-498e-91a2-2052c69da29c" containerName="test-curl-7301-97f2b8" containerID="cri-o://b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677" gracePeriod=2 Oct 30 22:02:21.328999 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:21.328960 5914 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7301-97f2b8\" is forbidden: User \"system:node:ip-10-0-142-119.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-119.us-west-1.compute.internal' and this object" podUID="1f8d30be-41d8-498e-91a2-2052c69da29c" pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:23.514161 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:23.514128 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:23.544681 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:23.544604 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp\" (UniqueName: \"kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp\") pod \"1f8d30be-41d8-498e-91a2-2052c69da29c\" (UID: \"1f8d30be-41d8-498e-91a2-2052c69da29c\") " Oct 30 22:02:23.546928 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:23.546900 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp" pod "1f8d30be-41d8-498e-91a2-2052c69da29c" (UID: "1f8d30be-41d8-498e-91a2-2052c69da29c"). InnerVolumeSpecName "kube-api-access-sgnwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:23.645688 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:23.645655 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgnwp\" (UniqueName: \"kubernetes.io/projected/1f8d30be-41d8-498e-91a2-2052c69da29c-kube-api-access-sgnwp\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:24.216632 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.216594 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8d30be-41d8-498e-91a2-2052c69da29c" path="/var/lib/kubelet/pods/1f8d30be-41d8-498e-91a2-2052c69da29c/volumes" Oct 30 22:02:24.296387 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.296350 5914 generic.go:356] "Generic (PLEG): container finished" podID="1f8d30be-41d8-498e-91a2-2052c69da29c" containerID="b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677" exitCode=137 Oct 30 22:02:24.296570 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.296400 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7301-97f2b8" Oct 30 22:02:24.296570 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.296455 5914 scope.go:122] "RemoveContainer" containerID="b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677" Oct 30 22:02:24.305758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.305732 5914 scope.go:122] "RemoveContainer" containerID="b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677" Oct 30 22:02:24.306095 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:24.306072 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677\": container with ID starting with b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677 not found: ID does not exist" containerID="b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677" Oct 30 22:02:24.306151 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.306104 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677"} err="failed to get container status \"b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677\": rpc error: code = NotFound desc = could not find container \"b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677\": container with ID starting with b1469ebc76afd7f0c0b1667055a66f5313b87313504fe63b4eeb14f6236de677 not found: ID does not exist" Oct 30 22:02:24.516407 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.516323 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g"] Oct 30 22:02:24.516812 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.516803 5914 cpu_manager.go:414] "containerMap: removing container" podUID="1f8d30be-41d8-498e-91a2-2052c69da29c" containerName="test-curl-7301-97f2b8" Oct 30 22:02:24.516946 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.516813 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8d30be-41d8-498e-91a2-2052c69da29c" containerName="test-curl-7301-97f2b8" Oct 30 22:02:24.517027 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.517009 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8d30be-41d8-498e-91a2-2052c69da29c" containerName="test-curl-7301-97f2b8" Oct 30 22:02:24.521581 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.521561 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555270 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555236 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7n2t\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555426 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555286 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555426 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555393 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555497 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555436 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555493 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555519 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555608 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555552 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555665 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555650 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.555701 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.555688 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.625286 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.625243 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g"] Oct 30 22:02:24.656850 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.656818 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7n2t\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.656866 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.656921 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.656956 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.656973 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.656989 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657049 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657024 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657400 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657091 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657400 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657132 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657520 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657410 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657587 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657556 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657642 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657600 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657697 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657667 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.657785 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.657764 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.659582 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.659554 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.659824 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.659806 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.713662 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.713629 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.713806 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.713732 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7n2t\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t\") pod \"e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:24.835822 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:24.835739 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:25.260298 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:25.260271 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g"] Oct 30 22:02:25.262310 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:02:25.262280 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604cd98f_1899_430f_bb6d_58312da4193a.slice/crio-a7766e1bc33779426d860c99faab3edbd0541c9190357ef6fbe02d9dc3a0a759 WatchSource:0}: Error finding container a7766e1bc33779426d860c99faab3edbd0541c9190357ef6fbe02d9dc3a0a759: Status 404 returned error can't find the container with id a7766e1bc33779426d860c99faab3edbd0541c9190357ef6fbe02d9dc3a0a759 Oct 30 22:02:25.264767 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:25.264734 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:25.264844 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:25.264800 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:25.264906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:25.264850 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:25.304556 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:25.304399 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" event={"ID":"604cd98f-1899-430f-bb6d-58312da4193a","Type":"ContainerStarted","Data":"a7766e1bc33779426d860c99faab3edbd0541c9190357ef6fbe02d9dc3a0a759"} Oct 30 22:02:26.311312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:26.311271 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" event={"ID":"604cd98f-1899-430f-bb6d-58312da4193a","Type":"ContainerStarted","Data":"2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d"} Oct 30 22:02:26.836861 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:26.836822 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:26.838513 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:26.838489 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:02:26.838602 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:26.838546 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 22:02:27.836628 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:27.836593 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:02:27.837059 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:27.836652 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 22:02:28.836800 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:28.836760 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.837240 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:28.836817 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 22:02:29.270772 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:29.270745 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:02:29.301389 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:29.301357 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:02:29.837072 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:29.837034 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.837505 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:29.837093 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 22:02:30.836606 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:30.836571 5914 patch_prober.go:29] interesting pod/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.836792 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:30.836630 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 22:02:31.840805 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:31.840774 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:31.841233 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:31.841212 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:31.842082 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:31.842065 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:02:32.010555 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:32.010494 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podStartSLOduration=8.01048083 podStartE2EDuration="8.01048083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:26.498083525 +0000 UTC m=+2698.399947593" watchObservedRunningTime="2026-10-30 22:02:32.01048083 +0000 UTC m=+2703.912344894" Oct 30 22:02:32.825149 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:32.825113 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h"] Oct 30 22:02:32.930338 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:32.930300 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh"] Oct 30 22:02:32.972149 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:32.972115 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:02:37.826376 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:37.826272 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" podUID="749695c2-24eb-4ecd-b8cb-870239a8b36a" containerName="payload-processing" containerID="cri-o://55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2" gracePeriod=25 Oct 30 22:02:37.931160 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:37.931072 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" podUID="9549c05c-a3e7-4349-a48f-838c2617f602" containerName="payload-pre-processing" containerID="cri-o://2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755" gracePeriod=25 Oct 30 22:02:38.106558 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.106480 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:02:38.202866 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.202844 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:02:38.236749 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.236704 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume\") pod \"749695c2-24eb-4ecd-b8cb-870239a8b36a\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " Oct 30 22:02:38.236928 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.236762 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv\" (UniqueName: \"kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv\") pod \"749695c2-24eb-4ecd-b8cb-870239a8b36a\" (UID: \"749695c2-24eb-4ecd-b8cb-870239a8b36a\") " Oct 30 22:02:38.237172 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.237143 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume" pod "749695c2-24eb-4ecd-b8cb-870239a8b36a" (UID: "749695c2-24eb-4ecd-b8cb-870239a8b36a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:38.237318 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.237298 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749695c2-24eb-4ecd-b8cb-870239a8b36a-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.239204 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.239174 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv" pod "749695c2-24eb-4ecd-b8cb-870239a8b36a" (UID: "749695c2-24eb-4ecd-b8cb-870239a8b36a"). InnerVolumeSpecName "kube-api-access-4lpsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:38.337915 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.337872 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume\") pod \"9549c05c-a3e7-4349-a48f-838c2617f602\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " Oct 30 22:02:38.338077 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.337965 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7\" (UniqueName: \"kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7\") pod \"9549c05c-a3e7-4349-a48f-838c2617f602\" (UID: \"9549c05c-a3e7-4349-a48f-838c2617f602\") " Oct 30 22:02:38.338239 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.338217 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume" pod "9549c05c-a3e7-4349-a48f-838c2617f602" (UID: "9549c05c-a3e7-4349-a48f-838c2617f602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:38.338339 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.338318 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9549c05c-a3e7-4349-a48f-838c2617f602-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.338339 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.338336 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lpsv\" (UniqueName: \"kubernetes.io/projected/749695c2-24eb-4ecd-b8cb-870239a8b36a-kube-api-access-4lpsv\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.340392 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.340369 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7" pod "9549c05c-a3e7-4349-a48f-838c2617f602" (UID: "9549c05c-a3e7-4349-a48f-838c2617f602"). InnerVolumeSpecName "kube-api-access-khdv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:38.386743 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.386706 5914 generic.go:356] "Generic (PLEG): container finished" podID="749695c2-24eb-4ecd-b8cb-870239a8b36a" containerID="55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2" exitCode=0 Oct 30 22:02:38.386911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.386758 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" Oct 30 22:02:38.386911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.386796 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" event={"ID":"749695c2-24eb-4ecd-b8cb-870239a8b36a","Type":"ContainerDied","Data":"55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2"} Oct 30 22:02:38.386911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.386872 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h" event={"ID":"749695c2-24eb-4ecd-b8cb-870239a8b36a","Type":"ContainerDied","Data":"e3058b1ed686a8a6a4831654c91493c09cf498b24f5e3bdae2686e8d051025f0"} Oct 30 22:02:38.387065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.386934 5914 scope.go:122] "RemoveContainer" containerID="55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2" Oct 30 22:02:38.388255 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.388232 5914 generic.go:356] "Generic (PLEG): container finished" podID="9549c05c-a3e7-4349-a48f-838c2617f602" containerID="2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755" exitCode=0 Oct 30 22:02:38.388369 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.388279 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" Oct 30 22:02:38.388369 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.388324 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" event={"ID":"9549c05c-a3e7-4349-a48f-838c2617f602","Type":"ContainerDied","Data":"2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755"} Oct 30 22:02:38.388369 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.388365 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh" event={"ID":"9549c05c-a3e7-4349-a48f-838c2617f602","Type":"ContainerDied","Data":"82cbd13fd5486008db0f7cb4f281853dc63d995dce3d08f1abd579015e9ca2b0"} Oct 30 22:02:38.397504 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.397475 5914 scope.go:122] "RemoveContainer" containerID="55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2" Oct 30 22:02:38.397791 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:38.397769 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2\": container with ID starting with 55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2 not found: ID does not exist" containerID="55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2" Oct 30 22:02:38.397898 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.397799 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2"} err="failed to get container status \"55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2\": rpc error: code = NotFound desc = could not find container \"55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2\": container with ID starting with 55df885b8070087e0c85c2f78b5e4e66f5c12e10068b08a3849b802441e933c2 not found: ID does not exist" Oct 30 22:02:38.397898 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.397820 5914 scope.go:122] "RemoveContainer" containerID="2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755" Oct 30 22:02:38.406710 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.406690 5914 scope.go:122] "RemoveContainer" containerID="2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755" Oct 30 22:02:38.406998 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:38.406978 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755\": container with ID starting with 2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755 not found: ID does not exist" containerID="2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755" Oct 30 22:02:38.407054 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.407005 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755"} err="failed to get container status \"2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755\": rpc error: code = NotFound desc = could not find container \"2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755\": container with ID starting with 2b4c0cdcaf7e0d09a97f3c3172e5959d23a1d9478d63c1e8fba9616f45dd2755 not found: ID does not exist" Oct 30 22:02:38.439695 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.439659 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdv7\" (UniqueName: \"kubernetes.io/projected/9549c05c-a3e7-4349-a48f-838c2617f602-kube-api-access-khdv7\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.617070 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.617034 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h"] Oct 30 22:02:38.629799 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.629767 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-75c2acf6-695b87cb85-mxv8h"] Oct 30 22:02:38.739280 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.739190 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh"] Oct 30 22:02:38.745361 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:38.745333 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-75c2acf6-7754c5cf75-66nkh"] Oct 30 22:02:40.196492 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:40.196456 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv"] Oct 30 22:02:40.217523 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:40.217484 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749695c2-24eb-4ecd-b8cb-870239a8b36a" path="/var/lib/kubelet/pods/749695c2-24eb-4ecd-b8cb-870239a8b36a/volumes" Oct 30 22:02:40.217980 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:40.217960 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9549c05c-a3e7-4349-a48f-838c2617f602" path="/var/lib/kubelet/pods/9549c05c-a3e7-4349-a48f-838c2617f602/volumes" Oct 30 22:02:42.346438 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:42.346401 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:02:48.952662 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:48.952615 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:02:52.245834 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:52.245799 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj"] Oct 30 22:02:52.352142 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:52.352104 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2"] Oct 30 22:02:55.218504 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.218444 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" podUID="2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" containerName="main" containerID="cri-o://0c79c63ad6744f9f989d937fb184b51b23370186d30f15b727dbb8e5ca969fef" gracePeriod=45 Oct 30 22:02:55.480731 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.480654 5914 generic.go:356] "Generic (PLEG): container finished" podID="2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" containerID="0c79c63ad6744f9f989d937fb184b51b23370186d30f15b727dbb8e5ca969fef" exitCode=0 Oct 30 22:02:55.480897 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.480725 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" event={"ID":"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52","Type":"ContainerDied","Data":"0c79c63ad6744f9f989d937fb184b51b23370186d30f15b727dbb8e5ca969fef"} Oct 30 22:02:55.480897 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.480766 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" event={"ID":"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52","Type":"ContainerDied","Data":"aff5391c561edd98375176c7fb65eca63d21db3b1eb26aee04fc3b58861d8b32"} Oct 30 22:02:55.480897 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.480781 5914 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff5391c561edd98375176c7fb65eca63d21db3b1eb26aee04fc3b58861d8b32" Oct 30 22:02:55.484531 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.484511 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 22:02:55.533092 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533062 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache\") pod \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " Oct 30 22:02:55.533261 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533148 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs\") pod \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " Oct 30 22:02:55.533261 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533210 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir\") pod \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " Oct 30 22:02:55.533261 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533244 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home\") pod \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " Oct 30 22:02:55.533442 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533271 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm\") pod \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " Oct 30 22:02:55.533442 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533337 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache" pod "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" (UID: "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:55.533442 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533343 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg\" (UniqueName: \"kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg\") pod \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\" (UID: \"2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52\") " Oct 30 22:02:55.533610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533502 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir" pod "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" (UID: "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:55.533610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533577 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home" pod "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" (UID: "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:55.533914 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533860 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:55.533914 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533918 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:55.534289 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.533930 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:55.535723 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.535699 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm" pod "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" (UID: "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:55.535723 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.535706 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg" pod "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" (UID: "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52"). InnerVolumeSpecName "kube-api-access-4z4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:55.535862 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.535719 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs" pod "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" (UID: "2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:55.634595 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.634561 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:55.634595 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.634594 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:55.634848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:55.634613 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4pg\" (UniqueName: \"kubernetes.io/projected/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52-kube-api-access-4z4pg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:56.486174 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:56.486091 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv" Oct 30 22:02:56.652137 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:56.652097 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv"] Oct 30 22:02:56.698479 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:56.698443 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dbe6da/test-model-dbe6da-kserve-5ff89c556c-28jmv"] Oct 30 22:02:57.246684 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:57.246619 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerName="payload-processing" containerID="cri-o://44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392" gracePeriod=25 Oct 30 22:02:57.352697 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:57.352638 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerName="payload-pre-processing" containerID="cri-o://44b9215d4fcc5ab5c3be6fd207d77124432ac54f9c1bb9161dd3bc35e5d861ca" gracePeriod=25 Oct 30 22:02:57.818959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:57.818906 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn"] Oct 30 22:02:57.819406 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:57.819151 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" podUID="f1715934-01e0-41b2-a2d2-a16265815f8d" containerName="maas-api" containerID="cri-o://fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad" gracePeriod=30 Oct 30 22:02:58.083893 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.083816 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:02:58.165318 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.165279 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca\") pod \"f1715934-01e0-41b2-a2d2-a16265815f8d\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " Oct 30 22:02:58.165499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.165351 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2\" (UniqueName: \"kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2\") pod \"f1715934-01e0-41b2-a2d2-a16265815f8d\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " Oct 30 22:02:58.165499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.165379 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\") pod \"f1715934-01e0-41b2-a2d2-a16265815f8d\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " Oct 30 22:02:58.165499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.165446 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs\") pod \"f1715934-01e0-41b2-a2d2-a16265815f8d\" (UID: \"f1715934-01e0-41b2-a2d2-a16265815f8d\") " Oct 30 22:02:58.165749 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.165721 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca" pod "f1715934-01e0-41b2-a2d2-a16265815f8d" (UID: "f1715934-01e0-41b2-a2d2-a16265815f8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:58.167731 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.167705 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2" pod "f1715934-01e0-41b2-a2d2-a16265815f8d" (UID: "f1715934-01e0-41b2-a2d2-a16265815f8d"). InnerVolumeSpecName "kube-api-access-w4xd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:58.167731 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.167711 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs" pod "f1715934-01e0-41b2-a2d2-a16265815f8d" (UID: "f1715934-01e0-41b2-a2d2-a16265815f8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:58.167731 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.167724 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls" pod "f1715934-01e0-41b2-a2d2-a16265815f8d" (UID: "f1715934-01e0-41b2-a2d2-a16265815f8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:58.225453 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.225407 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" path="/var/lib/kubelet/pods/2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52/volumes" Oct 30 22:02:58.267700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.267608 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1715934-01e0-41b2-a2d2-a16265815f8d-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.267700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.267642 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4xd2\" (UniqueName: \"kubernetes.io/projected/f1715934-01e0-41b2-a2d2-a16265815f8d-kube-api-access-w4xd2\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.267700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.267660 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.267700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.267673 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1715934-01e0-41b2-a2d2-a16265815f8d-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:58.498810 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.498770 5914 generic.go:356] "Generic (PLEG): container finished" podID="f1715934-01e0-41b2-a2d2-a16265815f8d" containerID="fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad" exitCode=0 Oct 30 22:02:58.499034 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.498827 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" Oct 30 22:02:58.499034 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.498858 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" event={"ID":"f1715934-01e0-41b2-a2d2-a16265815f8d","Type":"ContainerDied","Data":"fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad"} Oct 30 22:02:58.499034 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.498935 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn" event={"ID":"f1715934-01e0-41b2-a2d2-a16265815f8d","Type":"ContainerDied","Data":"4c8822ba4fe19d54b682a09007c3d31cba1b886974336c01f9545fbc6d17060e"} Oct 30 22:02:58.499034 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.498959 5914 scope.go:122] "RemoveContainer" containerID="fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad" Oct 30 22:02:58.509383 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.509363 5914 scope.go:122] "RemoveContainer" containerID="fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad" Oct 30 22:02:58.509687 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:02:58.509663 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad\": container with ID starting with fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad not found: ID does not exist" containerID="fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad" Oct 30 22:02:58.509799 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.509698 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad"} err="failed to get container status \"fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad\": rpc error: code = NotFound desc = could not find container \"fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad\": container with ID starting with fe53a8cb927dd08d9043e996347da274bf7f0f1e497157121ff8247524c3bdad not found: ID does not exist" Oct 30 22:02:58.661638 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.661604 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn"] Oct 30 22:02:58.663292 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:02:58.663269 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-75c2acf6-5f5dc8bc79-tfvwn"] Oct 30 22:03:00.216148 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:00.216111 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1715934-01e0-41b2-a2d2-a16265815f8d" path="/var/lib/kubelet/pods/f1715934-01e0-41b2-a2d2-a16265815f8d/volumes" Oct 30 22:03:06.099785 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:06.099742 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:03:06.786576 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:06.786534 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:03:06.786576 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:06.786555 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:03:10.248730 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:10.248678 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" podUID="d6458e4c6fa120c378f995c71dc796f0" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:12.345346 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:03:12.345303 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:03:16.785828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:16.785785 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:03:16.785828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:16.785802 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:03:17.514851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.514778 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 22:03:17.607797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.607635 5914 generic.go:356] "Generic (PLEG): container finished" podID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerID="44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392" exitCode=0 Oct 30 22:03:17.607797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.607696 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" Oct 30 22:03:17.607797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.607712 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" event={"ID":"48a80e15-dc93-46fd-ad5d-7bcbb1302717","Type":"ContainerDied","Data":"44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392"} Oct 30 22:03:17.607797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.607778 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj" event={"ID":"48a80e15-dc93-46fd-ad5d-7bcbb1302717","Type":"ContainerDied","Data":"07cf77b2d48a26339edaed015ece290ef42dd53ecdd3a45f1c012b74f2e61580"} Oct 30 22:03:17.608083 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.607808 5914 scope.go:122] "RemoveContainer" containerID="44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392" Oct 30 22:03:17.609195 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.609167 5914 generic.go:356] "Generic (PLEG): container finished" podID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerID="44b9215d4fcc5ab5c3be6fd207d77124432ac54f9c1bb9161dd3bc35e5d861ca" exitCode=0 Oct 30 22:03:17.609328 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.609238 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" event={"ID":"9c06fe3d-e407-4403-936d-ece9e1e72116","Type":"ContainerDied","Data":"44b9215d4fcc5ab5c3be6fd207d77124432ac54f9c1bb9161dd3bc35e5d861ca"} Oct 30 22:03:17.623016 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.622940 5914 scope.go:122] "RemoveContainer" containerID="44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392" Oct 30 22:03:17.623201 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:03:17.623178 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392\": container with ID starting with 44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392 not found: ID does not exist" containerID="44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392" Oct 30 22:03:17.623266 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.623213 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392"} err="failed to get container status \"44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392\": rpc error: code = NotFound desc = could not find container \"44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392\": container with ID starting with 44228bd0b244cc35ad8bd4b9e7bb30cba0f85cd1cd87ea6094ee07eda90b3392 not found: ID does not exist" Oct 30 22:03:17.626104 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.626086 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 22:03:17.674978 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.674942 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume\") pod \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " Oct 30 22:03:17.674978 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.674978 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6\" (UniqueName: \"kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6\") pod \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\" (UID: \"48a80e15-dc93-46fd-ad5d-7bcbb1302717\") " Oct 30 22:03:17.675325 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.675299 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume" pod "48a80e15-dc93-46fd-ad5d-7bcbb1302717" (UID: "48a80e15-dc93-46fd-ad5d-7bcbb1302717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:17.675458 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.675438 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48a80e15-dc93-46fd-ad5d-7bcbb1302717-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:17.677369 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.677346 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6" pod "48a80e15-dc93-46fd-ad5d-7bcbb1302717" (UID: "48a80e15-dc93-46fd-ad5d-7bcbb1302717"). InnerVolumeSpecName "kube-api-access-csld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:17.776605 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.776511 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume\") pod \"9c06fe3d-e407-4403-936d-ece9e1e72116\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " Oct 30 22:03:17.776605 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.776589 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l\" (UniqueName: \"kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l\") pod \"9c06fe3d-e407-4403-936d-ece9e1e72116\" (UID: \"9c06fe3d-e407-4403-936d-ece9e1e72116\") " Oct 30 22:03:17.776949 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.776924 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume" pod "9c06fe3d-e407-4403-936d-ece9e1e72116" (UID: "9c06fe3d-e407-4403-936d-ece9e1e72116"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:17.777038 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.777025 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c06fe3d-e407-4403-936d-ece9e1e72116-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:17.777085 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.777040 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csld6\" (UniqueName: \"kubernetes.io/projected/48a80e15-dc93-46fd-ad5d-7bcbb1302717-kube-api-access-csld6\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:17.778935 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.778910 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l" pod "9c06fe3d-e407-4403-936d-ece9e1e72116" (UID: "9c06fe3d-e407-4403-936d-ece9e1e72116"). InnerVolumeSpecName "kube-api-access-z652l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:17.878501 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:17.878463 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z652l\" (UniqueName: \"kubernetes.io/projected/9c06fe3d-e407-4403-936d-ece9e1e72116-kube-api-access-z652l\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:18.027513 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.027429 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj"] Oct 30 22:03:18.031206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.031178 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e9aa8e-7f96bdc687-sdcxj"] Oct 30 22:03:18.216461 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.216426 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" path="/var/lib/kubelet/pods/48a80e15-dc93-46fd-ad5d-7bcbb1302717/volumes" Oct 30 22:03:18.616830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.616804 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" Oct 30 22:03:18.617033 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.616801 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2" event={"ID":"9c06fe3d-e407-4403-936d-ece9e1e72116","Type":"ContainerDied","Data":"252bf84c762bf4ed06d4ae98cf5b25d4e875b6f08755313372e66b87e574dafb"} Oct 30 22:03:18.617033 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.616945 5914 scope.go:122] "RemoveContainer" containerID="44b9215d4fcc5ab5c3be6fd207d77124432ac54f9c1bb9161dd3bc35e5d861ca" Oct 30 22:03:18.780059 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.780014 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2"] Oct 30 22:03:18.829487 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:18.829452 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e9aa8e-56dbb478f5-qszc2"] Oct 30 22:03:20.215873 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:20.215838 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" path="/var/lib/kubelet/pods/9c06fe3d-e407-4403-936d-ece9e1e72116/volumes" Oct 30 22:03:27.966505 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:27.966469 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:03:40.196278 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:40.196243 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl"] Oct 30 22:03:42.346052 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:03:42.346015 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:03:45.196749 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.196698 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" podUID="4a01a558-919c-449b-861b-aed401f09b25" containerName="payload-processing" containerID="cri-o://ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd" gracePeriod=25 Oct 30 22:03:45.442239 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.442214 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 22:03:45.563281 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.563188 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx\" (UniqueName: \"kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx\") pod \"4a01a558-919c-449b-861b-aed401f09b25\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " Oct 30 22:03:45.563281 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.563254 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume\") pod \"4a01a558-919c-449b-861b-aed401f09b25\" (UID: \"4a01a558-919c-449b-861b-aed401f09b25\") " Oct 30 22:03:45.564048 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.564012 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume" pod "4a01a558-919c-449b-861b-aed401f09b25" (UID: "4a01a558-919c-449b-861b-aed401f09b25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:45.565848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.565821 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx" pod "4a01a558-919c-449b-861b-aed401f09b25" (UID: "4a01a558-919c-449b-861b-aed401f09b25"). InnerVolumeSpecName "kube-api-access-ttqsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:45.665288 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.665256 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttqsx\" (UniqueName: \"kubernetes.io/projected/4a01a558-919c-449b-861b-aed401f09b25-kube-api-access-ttqsx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:45.665288 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.665284 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a01a558-919c-449b-861b-aed401f09b25-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:45.757251 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.757212 5914 generic.go:356] "Generic (PLEG): container finished" podID="4a01a558-919c-449b-861b-aed401f09b25" containerID="ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd" exitCode=0 Oct 30 22:03:45.757434 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.757261 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" Oct 30 22:03:45.757434 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.757284 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" event={"ID":"4a01a558-919c-449b-861b-aed401f09b25","Type":"ContainerDied","Data":"ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd"} Oct 30 22:03:45.757434 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.757326 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl" event={"ID":"4a01a558-919c-449b-861b-aed401f09b25","Type":"ContainerDied","Data":"752a425809978e34f0b9c803208c0086e6da5b310e50147e686008205360ad4c"} Oct 30 22:03:45.757434 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.757348 5914 scope.go:122] "RemoveContainer" containerID="ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd" Oct 30 22:03:45.767153 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.767130 5914 scope.go:122] "RemoveContainer" containerID="ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd" Oct 30 22:03:45.767439 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:03:45.767416 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd\": container with ID starting with ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd not found: ID does not exist" containerID="ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd" Oct 30 22:03:45.767493 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.767450 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd"} err="failed to get container status \"ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd\": rpc error: code = NotFound desc = could not find container \"ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd\": container with ID starting with ea45575099efdedc11c0e34b648bbab6da2454087a9d9ecc3f995231d11764bd not found: ID does not exist" Oct 30 22:03:45.920332 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.920298 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl"] Oct 30 22:03:45.926592 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:45.926567 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dbe6da-db659d675-4ddhl"] Oct 30 22:03:46.215615 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:46.215534 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a01a558-919c-449b-861b-aed401f09b25" path="/var/lib/kubelet/pods/4a01a558-919c-449b-861b-aed401f09b25/volumes" Oct 30 22:03:53.773863 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:53.773823 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j"] Oct 30 22:03:53.776359 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:53.774059 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" podUID="20589f2d-3051-4314-b71d-4f2fe21f5b7b" containerName="maas-api" containerID="cri-o://2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53" gracePeriod=30 Oct 30 22:03:54.032731 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.032654 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 22:03:54.153641 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.153599 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6\" (UniqueName: \"kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6\") pod \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " Oct 30 22:03:54.153839 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.153691 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\") pod \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " Oct 30 22:03:54.153922 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.153858 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs\") pod \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " Oct 30 22:03:54.153984 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.153938 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca\") pod \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\" (UID: \"20589f2d-3051-4314-b71d-4f2fe21f5b7b\") " Oct 30 22:03:54.172433 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.159562 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca" pod "20589f2d-3051-4314-b71d-4f2fe21f5b7b" (UID: "20589f2d-3051-4314-b71d-4f2fe21f5b7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:54.172433 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.172111 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6" pod "20589f2d-3051-4314-b71d-4f2fe21f5b7b" (UID: "20589f2d-3051-4314-b71d-4f2fe21f5b7b"). InnerVolumeSpecName "kube-api-access-9jdm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:54.174509 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.174457 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs" pod "20589f2d-3051-4314-b71d-4f2fe21f5b7b" (UID: "20589f2d-3051-4314-b71d-4f2fe21f5b7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:54.179454 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.179423 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls" pod "20589f2d-3051-4314-b71d-4f2fe21f5b7b" (UID: "20589f2d-3051-4314-b71d-4f2fe21f5b7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:54.257828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.257787 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:54.257828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.257824 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20589f2d-3051-4314-b71d-4f2fe21f5b7b-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:54.258071 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.257838 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jdm6\" (UniqueName: \"kubernetes.io/projected/20589f2d-3051-4314-b71d-4f2fe21f5b7b-kube-api-access-9jdm6\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:54.258071 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.257852 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20589f2d-3051-4314-b71d-4f2fe21f5b7b-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:54.806311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.806274 5914 generic.go:356] "Generic (PLEG): container finished" podID="20589f2d-3051-4314-b71d-4f2fe21f5b7b" containerID="2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53" exitCode=0 Oct 30 22:03:54.806754 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.806342 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" Oct 30 22:03:54.806754 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.806363 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" event={"ID":"20589f2d-3051-4314-b71d-4f2fe21f5b7b","Type":"ContainerDied","Data":"2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53"} Oct 30 22:03:54.806754 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.806425 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j" event={"ID":"20589f2d-3051-4314-b71d-4f2fe21f5b7b","Type":"ContainerDied","Data":"ee2e7c3daa3613c3f08af252a95ffa3674df8fc95593d2673affe964ae8d56b3"} Oct 30 22:03:54.806754 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.806454 5914 scope.go:122] "RemoveContainer" containerID="2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53" Oct 30 22:03:54.815675 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.815659 5914 scope.go:122] "RemoveContainer" containerID="2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53" Oct 30 22:03:54.816083 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:03:54.816063 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53\": container with ID starting with 2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53 not found: ID does not exist" containerID="2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53" Oct 30 22:03:54.816152 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.816092 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53"} err="failed to get container status \"2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53\": rpc error: code = NotFound desc = could not find container \"2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53\": container with ID starting with 2e693ade5902be6c84adbfbf14b24b9c7ebf615c7ea658b93b2fa990ad71ef53 not found: ID does not exist" Oct 30 22:03:54.972406 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:54.972370 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j"] Oct 30 22:03:55.021805 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:55.021770 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dbe6da-6f8b445bb9-2jx7j"] Oct 30 22:03:56.216066 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:03:56.216036 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20589f2d-3051-4314-b71d-4f2fe21f5b7b" path="/var/lib/kubelet/pods/20589f2d-3051-4314-b71d-4f2fe21f5b7b/volumes" Oct 30 22:04:00.563237 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:00.563202 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:04:09.444875 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.444792 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd"] Oct 30 22:04:09.445300 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.445023 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" containerID="cri-o://51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092" gracePeriod=30 Oct 30 22:04:09.461517 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.461474 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g"] Oct 30 22:04:09.461796 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.461747 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" containerID="cri-o://2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d" gracePeriod=30 Oct 30 22:04:09.485544 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.485510 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z"] Oct 30 22:04:09.485732 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.485711 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" containerID="cri-o://f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64" gracePeriod=30 Oct 30 22:04:09.493974 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.493937 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2"] Oct 30 22:04:09.494162 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:09.494137 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" containerID="cri-o://167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab" gracePeriod=30 Oct 30 22:04:12.344992 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:12.344949 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:04:14.800996 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.800958 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:04:14.849255 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.849227 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:04:14.852542 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.852520 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:04:14.856505 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.856480 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:04:14.869782 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.869743 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.869999 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.869831 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.869999 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.869953 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870002 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870037 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870047 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.870131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870082 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870127 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870399 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870190 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870399 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870283 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.870399 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870386 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.870605 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870223 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data\") pod \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\" (UID: \"8b36b4f0-2032-45ef-9d3e-50e269b84dd0\") " Oct 30 22:04:14.870718 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.870656 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.871295 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.871269 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.871295 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.871300 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.871480 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.871316 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.871480 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.871332 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.871689 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.871659 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.872480 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.872453 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.872636 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.872617 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.872720 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.872663 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "kube-api-access-zl2xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.872835 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.872819 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy" pod "8b36b4f0-2032-45ef-9d3e-50e269b84dd0" (UID: "8b36b4f0-2032-45ef-9d3e-50e269b84dd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.912734 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.912699 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5"] Oct 30 22:04:14.921274 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.921235 5914 generic.go:356] "Generic (PLEG): container finished" podID="604cd98f-1899-430f-bb6d-58312da4193a" containerID="2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d" exitCode=0 Oct 30 22:04:14.921475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.921352 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" Oct 30 22:04:14.921475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.921367 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" event={"ID":"604cd98f-1899-430f-bb6d-58312da4193a","Type":"ContainerDied","Data":"2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d"} Oct 30 22:04:14.921475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.921422 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g" event={"ID":"604cd98f-1899-430f-bb6d-58312da4193a","Type":"ContainerDied","Data":"a7766e1bc33779426d860c99faab3edbd0541c9190357ef6fbe02d9dc3a0a759"} Oct 30 22:04:14.921475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.921451 5914 scope.go:122] "RemoveContainer" containerID="2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d" Oct 30 22:04:14.923573 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.923545 5914 generic.go:356] "Generic (PLEG): container finished" podID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerID="167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab" exitCode=0 Oct 30 22:04:14.923703 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.923602 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" Oct 30 22:04:14.923703 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.923650 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" event={"ID":"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb","Type":"ContainerDied","Data":"167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab"} Oct 30 22:04:14.923790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.923732 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2" event={"ID":"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb","Type":"ContainerDied","Data":"6484b395b5a9c520a2309cc4d0002f0e14e5aa2ef9695ee7a353f3cd3234368f"} Oct 30 22:04:14.924971 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.924950 5914 generic.go:356] "Generic (PLEG): container finished" podID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerID="51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092" exitCode=0 Oct 30 22:04:14.925088 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.924988 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" Oct 30 22:04:14.925088 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.925022 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" event={"ID":"b21043b4-66b0-4258-8cda-ee9e95f8ca59","Type":"ContainerDied","Data":"51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092"} Oct 30 22:04:14.925088 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.925065 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd" event={"ID":"b21043b4-66b0-4258-8cda-ee9e95f8ca59","Type":"ContainerDied","Data":"9b797114ad210a43ac4b76ca0712390b3a41ef7218f66d8a672728fa83ee634e"} Oct 30 22:04:14.926238 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.926218 5914 generic.go:356] "Generic (PLEG): container finished" podID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerID="f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64" exitCode=0 Oct 30 22:04:14.926332 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.926257 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" Oct 30 22:04:14.926332 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.926257 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" event={"ID":"8b36b4f0-2032-45ef-9d3e-50e269b84dd0","Type":"ContainerDied","Data":"f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64"} Oct 30 22:04:14.926332 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.926299 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z" event={"ID":"8b36b4f0-2032-45ef-9d3e-50e269b84dd0","Type":"ContainerDied","Data":"f722b4634b23a7c57cb2cb3fa76603010a2b9298866eb00c6b39bfab3947be8e"} Oct 30 22:04:14.933377 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.933357 5914 scope.go:122] "RemoveContainer" containerID="2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d" Oct 30 22:04:14.933807 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:14.933782 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d\": container with ID starting with 2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d not found: ID does not exist" containerID="2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d" Oct 30 22:04:14.933939 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.933821 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d"} err="failed to get container status \"2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d\": rpc error: code = NotFound desc = could not find container \"2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d\": container with ID starting with 2193eb22e0fbb02cf8c13149654825d1cd4a90772391df87382ff339eb6eae3d not found: ID does not exist" Oct 30 22:04:14.933939 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.933846 5914 scope.go:122] "RemoveContainer" containerID="167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab" Oct 30 22:04:14.942778 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.942759 5914 scope.go:122] "RemoveContainer" containerID="167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab" Oct 30 22:04:14.943126 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:14.943105 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab\": container with ID starting with 167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab not found: ID does not exist" containerID="167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab" Oct 30 22:04:14.943218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.943141 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab"} err="failed to get container status \"167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab\": rpc error: code = NotFound desc = could not find container \"167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab\": container with ID starting with 167db82f8994151e3769211561c6e49eea59aa6ac4e490a9c2836aa20a2ff5ab not found: ID does not exist" Oct 30 22:04:14.943218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.943165 5914 scope.go:122] "RemoveContainer" containerID="51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092" Oct 30 22:04:14.951441 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.951419 5914 scope.go:122] "RemoveContainer" containerID="51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092" Oct 30 22:04:14.951727 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:14.951705 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092\": container with ID starting with 51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092 not found: ID does not exist" containerID="51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092" Oct 30 22:04:14.951794 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.951737 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092"} err="failed to get container status \"51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092\": rpc error: code = NotFound desc = could not find container \"51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092\": container with ID starting with 51de280a426c3c6c57d47f2174ad235f8e90c61954510efff033c85415097092 not found: ID does not exist" Oct 30 22:04:14.951794 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.951754 5914 scope.go:122] "RemoveContainer" containerID="f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64" Oct 30 22:04:14.960251 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.960230 5914 scope.go:122] "RemoveContainer" containerID="f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64" Oct 30 22:04:14.960529 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:14.960507 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64\": container with ID starting with f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64 not found: ID does not exist" containerID="f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64" Oct 30 22:04:14.960618 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.960537 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64"} err="failed to get container status \"f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64\": rpc error: code = NotFound desc = could not find container \"f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64\": container with ID starting with f8104144d16decf2742b95fec7b9f0c4e94e24362b9661b20eeb5688b53f9d64 not found: ID does not exist" Oct 30 22:04:14.972579 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972550 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.972756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972598 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.972756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972623 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.972756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972659 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.972756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972680 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.972756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972700 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.972756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972740 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972821 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972854 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972924 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972959 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.972989 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973023 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973056 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973064 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973096 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973133 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973162 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973191 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973226 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973264 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973293 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973322 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973378 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973475 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973721 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.973337 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.974225 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.974574 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs\") pod \"604cd98f-1899-430f-bb6d-58312da4193a\" (UID: \"604cd98f-1899-430f-bb6d-58312da4193a\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.974626 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket\") pod \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\" (UID: \"9ad69e28-4b1e-4aa5-911e-f7456dbc36fb\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.974664 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs\") pod \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\" (UID: \"b21043b4-66b0-4258-8cda-ee9e95f8ca59\") " Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.974738 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975228 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975251 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975268 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975284 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975298 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2xd\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-kube-api-access-zl2xd\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975315 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.975329 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b36b4f0-2032-45ef-9d3e-50e269b84dd0-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.976277 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.976465 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.976945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.976856 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.976923 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.976932 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.977006 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.977122 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.977345 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.978058 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.978992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.978120 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.980479 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980450 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.980595 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980456 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.980595 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980554 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.980595 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980568 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.980595 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980581 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "kube-api-access-cqh28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.980778 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980624 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.980778 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980471 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:14.980848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.980831 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:14.981702 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.981681 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t" pod "604cd98f-1899-430f-bb6d-58312da4193a" (UID: "604cd98f-1899-430f-bb6d-58312da4193a"). InnerVolumeSpecName "kube-api-access-f7n2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.981831 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.981808 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.981831 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.981825 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token" pod "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" (UID: "9ad69e28-4b1e-4aa5-911e-f7456dbc36fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:14.982104 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:14.982085 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw" pod "b21043b4-66b0-4258-8cda-ee9e95f8ca59" (UID: "b21043b4-66b0-4258-8cda-ee9e95f8ca59"). InnerVolumeSpecName "kube-api-access-gthrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076533 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076560 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076569 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076580 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076589 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076600 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076608 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.076614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076617 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqh28\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-kube-api-access-cqh28\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076629 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076637 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076645 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7n2t\" (UniqueName: \"kubernetes.io/projected/604cd98f-1899-430f-bb6d-58312da4193a-kube-api-access-f7n2t\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076653 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076661 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076671 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076679 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076688 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076696 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/604cd98f-1899-430f-bb6d-58312da4193a-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076705 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076712 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076721 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthrw\" (UniqueName: \"kubernetes.io/projected/b21043b4-66b0-4258-8cda-ee9e95f8ca59-kube-api-access-gthrw\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076729 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076738 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076746 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b21043b4-66b0-4258-8cda-ee9e95f8ca59-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076754 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/604cd98f-1899-430f-bb6d-58312da4193a-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.077003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.076761 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/604cd98f-1899-430f-bb6d-58312da4193a-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:15.500381 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.500345 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z"] Oct 30 22:04:15.505190 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.505164 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-g2r7z"] Oct 30 22:04:15.733932 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.733826 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2"] Oct 30 22:04:15.737519 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.737485 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-w72x2"] Oct 30 22:04:15.849069 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.849032 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g"] Oct 30 22:04:15.855667 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.855640 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-r2p5g"] Oct 30 22:04:15.963959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.963925 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd"] Oct 30 22:04:15.972570 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:15.972545 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c2acf6-openshift-default-74d4bc544f-tlbwd"] Oct 30 22:04:16.215942 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:16.215904 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604cd98f-1899-430f-bb6d-58312da4193a" path="/var/lib/kubelet/pods/604cd98f-1899-430f-bb6d-58312da4193a/volumes" Oct 30 22:04:16.216354 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:16.216341 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" path="/var/lib/kubelet/pods/8b36b4f0-2032-45ef-9d3e-50e269b84dd0/volumes" Oct 30 22:04:16.216732 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:16.216720 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" path="/var/lib/kubelet/pods/9ad69e28-4b1e-4aa5-911e-f7456dbc36fb/volumes" Oct 30 22:04:16.217482 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:16.217467 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" path="/var/lib/kubelet/pods/b21043b4-66b0-4258-8cda-ee9e95f8ca59/volumes" Oct 30 22:04:19.080688 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:19.080656 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75"] Oct 30 22:04:29.938894 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:29.938825 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" containerID="cri-o://d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3" gracePeriod=45 Oct 30 22:04:30.203267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.203206 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 22:04:30.324770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.324737 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt\" (UniqueName: \"kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt\") pod \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " Oct 30 22:04:30.325023 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.324826 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs\") pod \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " Oct 30 22:04:30.325023 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.324865 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm\") pod \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " Oct 30 22:04:30.325023 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.324899 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir\") pod \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " Oct 30 22:04:30.325023 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.324930 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache\") pod \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " Oct 30 22:04:30.325023 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.324965 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home\") pod \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\" (UID: \"c33b0585-3495-40c3-a1bd-50a7b5f3bff4\") " Oct 30 22:04:30.325317 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.325226 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home" pod "c33b0585-3495-40c3-a1bd-50a7b5f3bff4" (UID: "c33b0585-3495-40c3-a1bd-50a7b5f3bff4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.325317 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.325255 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache" pod "c33b0585-3495-40c3-a1bd-50a7b5f3bff4" (UID: "c33b0585-3495-40c3-a1bd-50a7b5f3bff4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.325317 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.325233 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir" pod "c33b0585-3495-40c3-a1bd-50a7b5f3bff4" (UID: "c33b0585-3495-40c3-a1bd-50a7b5f3bff4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.325492 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.325461 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.325492 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.325484 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.325603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.325499 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.327343 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.327313 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm" pod "c33b0585-3495-40c3-a1bd-50a7b5f3bff4" (UID: "c33b0585-3495-40c3-a1bd-50a7b5f3bff4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:30.327343 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.327327 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs" pod "c33b0585-3495-40c3-a1bd-50a7b5f3bff4" (UID: "c33b0585-3495-40c3-a1bd-50a7b5f3bff4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:30.327489 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.327393 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt" pod "c33b0585-3495-40c3-a1bd-50a7b5f3bff4" (UID: "c33b0585-3495-40c3-a1bd-50a7b5f3bff4"). InnerVolumeSpecName "kube-api-access-htrgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:30.426456 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.426424 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.426456 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.426450 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrgt\" (UniqueName: \"kubernetes.io/projected/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-kube-api-access-htrgt\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:30.426456 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:30.426460 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33b0585-3495-40c3-a1bd-50a7b5f3bff4-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:31.019241 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.019204 5914 generic.go:356] "Generic (PLEG): container finished" podID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerID="d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3" exitCode=0 Oct 30 22:04:31.019645 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.019248 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" Oct 30 22:04:31.019645 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.019286 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" event={"ID":"c33b0585-3495-40c3-a1bd-50a7b5f3bff4","Type":"ContainerDied","Data":"d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3"} Oct 30 22:04:31.019645 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.019334 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5" event={"ID":"c33b0585-3495-40c3-a1bd-50a7b5f3bff4","Type":"ContainerDied","Data":"f6fdd69011ac15dd3323a1814bde89c00cdb601d60a8d86390dfca4129733ddd"} Oct 30 22:04:31.019645 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.019356 5914 scope.go:122] "RemoveContainer" containerID="d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3" Oct 30 22:04:31.028361 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.028346 5914 scope.go:122] "RemoveContainer" containerID="d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3" Oct 30 22:04:31.028627 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:31.028612 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3\": container with ID starting with d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3 not found: ID does not exist" containerID="d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3" Oct 30 22:04:31.028738 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.028635 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3"} err="failed to get container status \"d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3\": rpc error: code = NotFound desc = could not find container \"d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3\": container with ID starting with d741eac4e8c9f18f2f73be26a4a57ef9a82fd1dc0923a52ada95f63c0ac2f9b3 not found: ID does not exist" Oct 30 22:04:31.181558 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.181526 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5"] Oct 30 22:04:31.231042 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:31.231006 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-07fa8e/simulator-w4-07fa8e-kserve-79df475566-rpqv5"] Oct 30 22:04:32.215325 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:32.215288 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" path="/var/lib/kubelet/pods/c33b0585-3495-40c3-a1bd-50a7b5f3bff4/volumes" Oct 30 22:04:34.101501 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.101433 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" containerID="cri-o://b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db" gracePeriod=45 Oct 30 22:04:34.351758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.351696 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 22:04:34.465024 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.464987 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs\") pod \"8a117ec3-78ec-418e-aec2-f333f54d2935\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " Oct 30 22:04:34.465204 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465061 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir\") pod \"8a117ec3-78ec-418e-aec2-f333f54d2935\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " Oct 30 22:04:34.465204 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465094 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache\") pod \"8a117ec3-78ec-418e-aec2-f333f54d2935\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " Oct 30 22:04:34.465204 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465153 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm\") pod \"8a117ec3-78ec-418e-aec2-f333f54d2935\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " Oct 30 22:04:34.465328 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465202 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home\") pod \"8a117ec3-78ec-418e-aec2-f333f54d2935\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " Oct 30 22:04:34.465328 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465317 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg\" (UniqueName: \"kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg\") pod \"8a117ec3-78ec-418e-aec2-f333f54d2935\" (UID: \"8a117ec3-78ec-418e-aec2-f333f54d2935\") " Oct 30 22:04:34.465449 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465389 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir" pod "8a117ec3-78ec-418e-aec2-f333f54d2935" (UID: "8a117ec3-78ec-418e-aec2-f333f54d2935"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.465518 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465437 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache" pod "8a117ec3-78ec-418e-aec2-f333f54d2935" (UID: "8a117ec3-78ec-418e-aec2-f333f54d2935"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.465577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465542 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home" pod "8a117ec3-78ec-418e-aec2-f333f54d2935" (UID: "8a117ec3-78ec-418e-aec2-f333f54d2935"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.465708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465691 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.465708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465708 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.465824 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.465718 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.467517 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.467488 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg" pod "8a117ec3-78ec-418e-aec2-f333f54d2935" (UID: "8a117ec3-78ec-418e-aec2-f333f54d2935"). InnerVolumeSpecName "kube-api-access-wctpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:34.467517 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.467498 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm" pod "8a117ec3-78ec-418e-aec2-f333f54d2935" (UID: "8a117ec3-78ec-418e-aec2-f333f54d2935"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:34.467642 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.467543 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs" pod "8a117ec3-78ec-418e-aec2-f333f54d2935" (UID: "8a117ec3-78ec-418e-aec2-f333f54d2935"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:34.566753 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.566721 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a117ec3-78ec-418e-aec2-f333f54d2935-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.566753 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.566752 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wctpg\" (UniqueName: \"kubernetes.io/projected/8a117ec3-78ec-418e-aec2-f333f54d2935-kube-api-access-wctpg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.566981 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:34.566763 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a117ec3-78ec-418e-aec2-f333f54d2935-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:35.040425 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.040385 5914 generic.go:356] "Generic (PLEG): container finished" podID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerID="b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db" exitCode=0 Oct 30 22:04:35.040610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.040429 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" Oct 30 22:04:35.040610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.040472 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" event={"ID":"8a117ec3-78ec-418e-aec2-f333f54d2935","Type":"ContainerDied","Data":"b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db"} Oct 30 22:04:35.040610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.040513 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75" event={"ID":"8a117ec3-78ec-418e-aec2-f333f54d2935","Type":"ContainerDied","Data":"c2198d50b306945f94c06f9a7180911952d81c7bb3c4e4da7f6455a9c2e7809f"} Oct 30 22:04:35.040610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.040529 5914 scope.go:122] "RemoveContainer" containerID="b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db" Oct 30 22:04:35.049748 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.049729 5914 scope.go:122] "RemoveContainer" containerID="b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db" Oct 30 22:04:35.050081 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:35.050061 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db\": container with ID starting with b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db not found: ID does not exist" containerID="b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db" Oct 30 22:04:35.050140 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.050091 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db"} err="failed to get container status \"b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db\": rpc error: code = NotFound desc = could not find container \"b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db\": container with ID starting with b422ee5de8553dd07f804416ca50c38ebef30060df69083ce706677cb08425db not found: ID does not exist" Oct 30 22:04:35.202756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.202716 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75"] Oct 30 22:04:35.252397 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:35.252353 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-446ca2/premium-w2-446ca2-kserve-5499b9fd4-vdn75"] Oct 30 22:04:36.215242 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:36.215205 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" path="/var/lib/kubelet/pods/8a117ec3-78ec-418e-aec2-f333f54d2935/volumes" Oct 30 22:04:39.602965 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:39.602927 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:04:42.346627 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:42.346591 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:04:48.922299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:48.922262 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5"] Oct 30 22:04:48.922906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:48.922475 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" containerID="cri-o://72dbe157cef1ad9e0c84e8ec9e60dca1a4e25975337727246b3d46b0b750462d" gracePeriod=30 Oct 30 22:04:49.217641 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.217558 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k"] Oct 30 22:04:49.217793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.217737 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" podUID="7abed670-4638-47a4-acf3-d3eb0b2d7c58" containerName="maas-api" containerID="cri-o://0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f" gracePeriod=30 Oct 30 22:04:49.465639 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.465615 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 22:04:49.512391 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.512309 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh\" (UniqueName: \"kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh\") pod \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " Oct 30 22:04:49.512391 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.512387 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\") pod \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " Oct 30 22:04:49.512628 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.512432 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca\") pod \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " Oct 30 22:04:49.512628 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.512461 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs\") pod \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\" (UID: \"7abed670-4638-47a4-acf3-d3eb0b2d7c58\") " Oct 30 22:04:49.512830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.512803 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca" pod "7abed670-4638-47a4-acf3-d3eb0b2d7c58" (UID: "7abed670-4638-47a4-acf3-d3eb0b2d7c58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:49.514730 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.514703 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh" pod "7abed670-4638-47a4-acf3-d3eb0b2d7c58" (UID: "7abed670-4638-47a4-acf3-d3eb0b2d7c58"). InnerVolumeSpecName "kube-api-access-cnssh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:49.514730 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.514709 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs" pod "7abed670-4638-47a4-acf3-d3eb0b2d7c58" (UID: "7abed670-4638-47a4-acf3-d3eb0b2d7c58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:49.514902 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.514746 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls" pod "7abed670-4638-47a4-acf3-d3eb0b2d7c58" (UID: "7abed670-4638-47a4-acf3-d3eb0b2d7c58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:49.613445 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.613405 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:49.613445 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.613444 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnssh\" (UniqueName: \"kubernetes.io/projected/7abed670-4638-47a4-acf3-d3eb0b2d7c58-kube-api-access-cnssh\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:49.613658 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.613458 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7abed670-4638-47a4-acf3-d3eb0b2d7c58-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:49.613658 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:49.613470 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abed670-4638-47a4-acf3-d3eb0b2d7c58-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:50.120356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.120316 5914 generic.go:356] "Generic (PLEG): container finished" podID="7abed670-4638-47a4-acf3-d3eb0b2d7c58" containerID="0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f" exitCode=0 Oct 30 22:04:50.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.120365 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" Oct 30 22:04:50.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.120397 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" event={"ID":"7abed670-4638-47a4-acf3-d3eb0b2d7c58","Type":"ContainerDied","Data":"0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f"} Oct 30 22:04:50.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.120453 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k" event={"ID":"7abed670-4638-47a4-acf3-d3eb0b2d7c58","Type":"ContainerDied","Data":"eede64f306c1c02d21fcb48756371329cd0e53655e12f849c7db6a79ace8e7c2"} Oct 30 22:04:50.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.120477 5914 scope.go:122] "RemoveContainer" containerID="0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f" Oct 30 22:04:50.133472 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.133454 5914 scope.go:122] "RemoveContainer" containerID="0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f" Oct 30 22:04:50.133759 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:04:50.133740 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f\": container with ID starting with 0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f not found: ID does not exist" containerID="0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f" Oct 30 22:04:50.133805 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.133770 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f"} err="failed to get container status \"0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f\": rpc error: code = NotFound desc = could not find container \"0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f\": container with ID starting with 0e46d6bd6970aab88f4e6302523cdd44f797b29778f4530c3a257461571b7a1f not found: ID does not exist" Oct 30 22:04:50.282348 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.282318 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k"] Oct 30 22:04:50.285790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:50.285768 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-07fa8e-59c5bbbbf4-7sg4k"] Oct 30 22:04:52.216483 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:52.216448 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abed670-4638-47a4-acf3-d3eb0b2d7c58" path="/var/lib/kubelet/pods/7abed670-4638-47a4-acf3-d3eb0b2d7c58/volumes" Oct 30 22:04:54.143397 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.143365 5914 generic.go:356] "Generic (PLEG): container finished" podID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerID="72dbe157cef1ad9e0c84e8ec9e60dca1a4e25975337727246b3d46b0b750462d" exitCode=0 Oct 30 22:04:54.143780 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.143441 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" event={"ID":"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019","Type":"ContainerDied","Data":"72dbe157cef1ad9e0c84e8ec9e60dca1a4e25975337727246b3d46b0b750462d"} Oct 30 22:04:54.172200 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.172169 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 22:04:54.258756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258667 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.258756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258710 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.258756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258730 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258768 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258795 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258831 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258861 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258921 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.258978 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs\") pod \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\" (UID: \"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019\") " Oct 30 22:04:54.259436 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259098 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:54.259436 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259203 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:54.259436 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259298 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:54.259436 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259427 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.259663 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259446 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.259663 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259462 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.259663 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259473 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:54.259663 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.259587 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:54.261171 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.261144 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "kube-api-access-w7l8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:54.261310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.261202 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:54.261357 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.261309 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:54.261394 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.261361 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo" pod "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" (UID: "ee5bd682-1e8c-45d2-bda9-5d8a6ea96019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:54.360634 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.360601 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.360634 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.360632 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.360634 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.360643 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.360875 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.360651 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7l8v\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-kube-api-access-w7l8v\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.360875 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.360661 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:54.360875 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:54.360669 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:55.148967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:55.148939 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" Oct 30 22:04:55.149389 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:55.148981 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5" event={"ID":"ee5bd682-1e8c-45d2-bda9-5d8a6ea96019","Type":"ContainerDied","Data":"bac429499c061e0d19cc5457aa9de982c9cc36f427d87b51b4fa8bada4d6c848"} Oct 30 22:04:55.149389 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:55.149042 5914 scope.go:122] "RemoveContainer" containerID="72dbe157cef1ad9e0c84e8ec9e60dca1a4e25975337727246b3d46b0b750462d" Oct 30 22:04:55.365173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:55.365135 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5"] Oct 30 22:04:55.414768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:55.414677 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-446ca2-openshift-default-f46fcc757-8d9g5"] Oct 30 22:04:56.215593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:04:56.215554 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" path="/var/lib/kubelet/pods/ee5bd682-1e8c-45d2-bda9-5d8a6ea96019/volumes" Oct 30 22:05:00.578046 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:00.578008 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:05:08.043280 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:08.043238 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7"] Oct 30 22:05:08.092835 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:08.092798 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b2714a-j6bz7"] Oct 30 22:05:08.215898 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:08.215847 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c12d75c-5433-4ade-b30b-d5c114ff04ee" path="/var/lib/kubelet/pods/7c12d75c-5433-4ade-b30b-d5c114ff04ee/volumes" Oct 30 22:05:12.344437 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:05:12.344397 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:05:14.711333 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711295 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-zltbj"] Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711708 5914 cpu_manager.go:414] "containerMap: removing container" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711720 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711734 5914 cpu_manager.go:414] "containerMap: removing container" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerName="payload-processing" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711740 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerName="payload-processing" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711749 5914 cpu_manager.go:414] "containerMap: removing container" podUID="749695c2-24eb-4ecd-b8cb-870239a8b36a" containerName="payload-processing" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711755 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749695c2-24eb-4ecd-b8cb-870239a8b36a" containerName="payload-processing" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711765 5914 cpu_manager.go:414] "containerMap: removing container" podUID="7abed670-4638-47a4-acf3-d3eb0b2d7c58" containerName="maas-api" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711770 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abed670-4638-47a4-acf3-d3eb0b2d7c58" containerName="maas-api" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711778 5914 cpu_manager.go:414] "containerMap: removing container" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" Oct 30 22:05:14.711781 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711783 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711792 5914 cpu_manager.go:414] "containerMap: removing container" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711797 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711802 5914 cpu_manager.go:414] "containerMap: removing container" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerName="payload-pre-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711807 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerName="payload-pre-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711817 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711821 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711832 5914 cpu_manager.go:414] "containerMap: removing container" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711836 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711844 5914 cpu_manager.go:414] "containerMap: removing container" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711848 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711855 5914 cpu_manager.go:414] "containerMap: removing container" podUID="4a01a558-919c-449b-861b-aed401f09b25" containerName="payload-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711860 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a01a558-919c-449b-861b-aed401f09b25" containerName="payload-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711866 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711870 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711891 5914 cpu_manager.go:414] "containerMap: removing container" podUID="20589f2d-3051-4314-b71d-4f2fe21f5b7b" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711895 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20589f2d-3051-4314-b71d-4f2fe21f5b7b" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711904 5914 cpu_manager.go:414] "containerMap: removing container" podUID="2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711908 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711919 5914 cpu_manager.go:414] "containerMap: removing container" podUID="9549c05c-a3e7-4349-a48f-838c2617f602" containerName="payload-pre-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711923 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9549c05c-a3e7-4349-a48f-838c2617f602" containerName="payload-pre-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711932 5914 cpu_manager.go:414] "containerMap: removing container" podUID="f1715934-01e0-41b2-a2d2-a16265815f8d" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711936 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1715934-01e0-41b2-a2d2-a16265815f8d" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711984 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abed670-4638-47a4-acf3-d3eb0b2d7c58" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711993 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad69e28-4b1e-4aa5-911e-f7456dbc36fb" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.711999 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21043b4-66b0-4258-8cda-ee9e95f8ca59" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712006 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c06fe3d-e407-4403-936d-ece9e1e72116" containerName="payload-pre-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712013 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5bd682-1e8c-45d2-bda9-5d8a6ea96019" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712019 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b36b4f0-2032-45ef-9d3e-50e269b84dd0" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712025 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="604cd98f-1899-430f-bb6d-58312da4193a" containerName="istio-proxy" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712031 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a01a558-919c-449b-861b-aed401f09b25" containerName="payload-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712036 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33b0585-3495-40c3-a1bd-50a7b5f3bff4" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712042 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="20589f2d-3051-4314-b71d-4f2fe21f5b7b" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712047 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3cbc1e-e5ee-4d81-8eac-9ccd03ec6f52" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712053 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9549c05c-a3e7-4349-a48f-838c2617f602" containerName="payload-pre-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712058 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1715934-01e0-41b2-a2d2-a16265815f8d" containerName="maas-api" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712063 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a117ec3-78ec-418e-aec2-f333f54d2935" containerName="main" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712068 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a80e15-dc93-46fd-ad5d-7bcbb1302717" containerName="payload-processing" Oct 30 22:05:14.712157 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.712073 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="749695c2-24eb-4ecd-b8cb-870239a8b36a" containerName="payload-processing" Oct 30 22:05:14.716502 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.716485 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.753937 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.753905 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.754070 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.754040 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45wqg\" (UniqueName: \"kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.754121 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.754081 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.809759 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.809701 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:05:14.810111 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.810093 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ffhp5\"" Oct 30 22:05:14.810170 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.810137 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:05:14.810266 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.810252 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:05:14.846374 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.846344 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-zltbj"] Oct 30 22:05:14.854516 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.854491 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.854612 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.854565 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45wqg\" (UniqueName: \"kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.854612 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.854597 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.857169 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.857140 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.857169 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.857162 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:14.916665 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:14.916630 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45wqg\" (UniqueName: \"kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg\") pod \"maas-controller-58d6bcb5f-zltbj\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:15.029447 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:15.029353 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:15.302700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:15.302616 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:05:15.467912 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:15.467849 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-zltbj"] Oct 30 22:05:15.471099 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:05:15.471069 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59714cf_7528_4ece_a09e_3bad8db1bedb.slice/crio-09354ff5d0e6033d27c411bc83c88ea9e433d7a9ebc99b99878590b755b953fb WatchSource:0}: Error finding container 09354ff5d0e6033d27c411bc83c88ea9e433d7a9ebc99b99878590b755b953fb: Status 404 returned error can't find the container with id 09354ff5d0e6033d27c411bc83c88ea9e433d7a9ebc99b99878590b755b953fb Oct 30 22:05:15.472475 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:15.472453 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:05:16.250809 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:16.250768 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" event={"ID":"b59714cf-7528-4ece-a09e-3bad8db1bedb","Type":"ContainerStarted","Data":"09354ff5d0e6033d27c411bc83c88ea9e433d7a9ebc99b99878590b755b953fb"} Oct 30 22:05:17.257212 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:17.257179 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" event={"ID":"b59714cf-7528-4ece-a09e-3bad8db1bedb","Type":"ContainerStarted","Data":"b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3"} Oct 30 22:05:17.257614 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:17.257312 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:17.374341 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:17.374288 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" podStartSLOduration=2.42790527 podStartE2EDuration="3.374270792s" totalImagesPullingTime="946.365522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:17.373497309 +0000 UTC m=+2869.275361375" watchObservedRunningTime="2026-10-30 22:05:17.374270792 +0000 UTC m=+2869.276134858" Oct 30 22:05:28.265954 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:28.265921 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:30.186527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:30.186481 5914 scope.go:122] "RemoveContainer" containerID="0c79c63ad6744f9f989d937fb184b51b23370186d30f15b727dbb8e5ca969fef" Oct 30 22:05:42.177526 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:42.177438 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:05:42.344993 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:05:42.344949 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:05:48.298125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.298085 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9"] Oct 30 22:05:48.303131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.303111 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.347529 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.347498 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9"] Oct 30 22:05:48.351067 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.351050 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.363852 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.363833 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 22:05:48.364041 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.364023 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x8rmh\"" Oct 30 22:05:48.405981 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.405945 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9"] Oct 30 22:05:48.469529 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.469493 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9"] Oct 30 22:05:48.471440 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471411 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.471580 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471445 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.471580 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471479 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hckx\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.471580 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471570 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.471753 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471601 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.471811 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471743 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.471892 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471822 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.471892 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471863 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.472006 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471949 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.472006 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.471970 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.472108 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472030 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.472108 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472076 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.472528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472151 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.472528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472183 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.472528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472200 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.472528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472229 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqdj\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.472528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472286 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.472528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.472311 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573231 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573281 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.573311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573300 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573322 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573345 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573400 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573426 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573450 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573486 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqdj\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573519 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573550 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573592 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.573626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573618 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573745 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573761 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573771 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hckx\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573821 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573866 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573933 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.573963 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574030 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574174 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574175 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.574310 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574289 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.575152 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574336 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.575152 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574494 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.575152 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574539 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.575152 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.574917 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.576130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.576109 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.576359 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.576341 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.576419 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.576345 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.576801 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.576776 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.630934 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.630906 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.631146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.631119 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.631193 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.631174 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hckx\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-lwsc9\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.631229 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.631212 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqdj\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj\") pod \"maas-default-gateway-openshift-default-7f694df8bc-66hx9\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:48.663566 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.663539 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:48.915566 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:48.915534 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:49.087684 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.087662 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9"] Oct 30 22:05:49.090027 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:05:49.089995 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce3fca9_5560_447a_9934_6a571dc6367a.slice/crio-55bc0de7a4439f45a559a5f2e1072685739b2ff8fa800f94d9bd35bfa8cb6373 WatchSource:0}: Error finding container 55bc0de7a4439f45a559a5f2e1072685739b2ff8fa800f94d9bd35bfa8cb6373: Status 404 returned error can't find the container with id 55bc0de7a4439f45a559a5f2e1072685739b2ff8fa800f94d9bd35bfa8cb6373 Oct 30 22:05:49.092282 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.092246 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:49.092380 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.092315 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:49.092380 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.092345 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:49.291163 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.291130 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9"] Oct 30 22:05:49.292377 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:05:49.292350 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee23856_34e6_4110_8e8f_ae9d13253bcf.slice/crio-759cd932809be06836199d26a148a198e7ba36466a77d94188da86ec2f33e84b WatchSource:0}: Error finding container 759cd932809be06836199d26a148a198e7ba36466a77d94188da86ec2f33e84b: Status 404 returned error can't find the container with id 759cd932809be06836199d26a148a198e7ba36466a77d94188da86ec2f33e84b Oct 30 22:05:49.295338 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.295306 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:49.295422 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.295376 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:49.295422 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.295405 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:49.411868 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.411824 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" event={"ID":"bee23856-34e6-4110-8e8f-ae9d13253bcf","Type":"ContainerStarted","Data":"d878b24f4556539150ba57fc60b2fc69019dfa680c35256f676fe30cd4f6001b"} Oct 30 22:05:49.412399 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.411908 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" event={"ID":"bee23856-34e6-4110-8e8f-ae9d13253bcf","Type":"ContainerStarted","Data":"759cd932809be06836199d26a148a198e7ba36466a77d94188da86ec2f33e84b"} Oct 30 22:05:49.413732 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.413706 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" event={"ID":"dce3fca9-5560-447a-9934-6a571dc6367a","Type":"ContainerStarted","Data":"b93fab8d55833bf9b1321afc803d623ee9e9f9145ee89877d3b46ab00fe27b0f"} Oct 30 22:05:49.413928 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.413746 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" event={"ID":"dce3fca9-5560-447a-9934-6a571dc6367a","Type":"ContainerStarted","Data":"55bc0de7a4439f45a559a5f2e1072685739b2ff8fa800f94d9bd35bfa8cb6373"} Oct 30 22:05:49.528243 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.528184 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podStartSLOduration=1.528167687 podStartE2EDuration="1.528167687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:49.52787372 +0000 UTC m=+2901.429737797" watchObservedRunningTime="2026-10-30 22:05:49.528167687 +0000 UTC m=+2901.430031751" Oct 30 22:05:49.647830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.647774 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podStartSLOduration=1.64775564 podStartE2EDuration="1.64775564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:49.64622228 +0000 UTC m=+2901.548086345" watchObservedRunningTime="2026-10-30 22:05:49.64775564 +0000 UTC m=+2901.549619699" Oct 30 22:05:49.664726 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.664694 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:49.920970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:49.920236 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:50.665430 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:50.665398 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lwsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:05:50.665841 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:50.665450 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:05:50.752371 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:50.752324 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-zltbj"] Oct 30 22:05:50.752635 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:50.752611 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" podUID="b59714cf-7528-4ece-a09e-3bad8db1bedb" containerName="manager" containerID="cri-o://b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3" gracePeriod=30 Oct 30 22:05:50.919627 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:50.919221 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:50.919627 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:50.919284 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:51.131247 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.131220 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:51.132525 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.132266 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs\") pod \"b59714cf-7528-4ece-a09e-3bad8db1bedb\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " Oct 30 22:05:51.132525 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.132321 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg\" (UniqueName: \"kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg\") pod \"b59714cf-7528-4ece-a09e-3bad8db1bedb\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " Oct 30 22:05:51.132525 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.132343 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert\") pod \"b59714cf-7528-4ece-a09e-3bad8db1bedb\" (UID: \"b59714cf-7528-4ece-a09e-3bad8db1bedb\") " Oct 30 22:05:51.134462 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.134436 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs" pod "b59714cf-7528-4ece-a09e-3bad8db1bedb" (UID: "b59714cf-7528-4ece-a09e-3bad8db1bedb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:51.135479 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.135457 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg" pod "b59714cf-7528-4ece-a09e-3bad8db1bedb" (UID: "b59714cf-7528-4ece-a09e-3bad8db1bedb"). InnerVolumeSpecName "kube-api-access-45wqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:51.135560 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.135456 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert" pod "b59714cf-7528-4ece-a09e-3bad8db1bedb" (UID: "b59714cf-7528-4ece-a09e-3bad8db1bedb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:51.233830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.233726 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.233830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.233766 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45wqg\" (UniqueName: \"kubernetes.io/projected/b59714cf-7528-4ece-a09e-3bad8db1bedb-kube-api-access-45wqg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.233830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.233781 5914 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b59714cf-7528-4ece-a09e-3bad8db1bedb-webhook-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.435436 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.435396 5914 generic.go:356] "Generic (PLEG): container finished" podID="b59714cf-7528-4ece-a09e-3bad8db1bedb" containerID="b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3" exitCode=0 Oct 30 22:05:51.435622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.435456 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" Oct 30 22:05:51.435622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.435489 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" event={"ID":"b59714cf-7528-4ece-a09e-3bad8db1bedb","Type":"ContainerDied","Data":"b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3"} Oct 30 22:05:51.435622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.435543 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d6bcb5f-zltbj" event={"ID":"b59714cf-7528-4ece-a09e-3bad8db1bedb","Type":"ContainerDied","Data":"09354ff5d0e6033d27c411bc83c88ea9e433d7a9ebc99b99878590b755b953fb"} Oct 30 22:05:51.435622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.435572 5914 scope.go:122] "RemoveContainer" containerID="b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3" Oct 30 22:05:51.445846 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.445829 5914 scope.go:122] "RemoveContainer" containerID="b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3" Oct 30 22:05:51.446156 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:05:51.446135 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3\": container with ID starting with b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3 not found: ID does not exist" containerID="b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3" Oct 30 22:05:51.446200 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.446167 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3"} err="failed to get container status \"b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3\": rpc error: code = NotFound desc = could not find container \"b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3\": container with ID starting with b0a0f46006541153ab6217801a6cec4d99bcb55a4e6be7a8108fb50ba6ed25e3 not found: ID does not exist" Oct 30 22:05:51.616963 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.616357 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-zltbj"] Oct 30 22:05:51.627554 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.627498 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-zltbj"] Oct 30 22:05:51.663959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.663923 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lwsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:05:51.664123 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.663997 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:05:51.920907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.920321 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:51.920907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:51.920388 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:52.224607 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:52.224521 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59714cf-7528-4ece-a09e-3bad8db1bedb" path="/var/lib/kubelet/pods/b59714cf-7528-4ece-a09e-3bad8db1bedb/volumes" Oct 30 22:05:52.664865 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:52.664825 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lwsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:05:52.665084 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:52.664902 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:05:52.919965 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:52.919347 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:52.919965 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:52.919421 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:53.664193 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:53.664158 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lwsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:05:53.664598 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:53.664213 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:05:53.919968 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:53.919497 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:53.919968 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:53.919590 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:54.664114 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:54.664075 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lwsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:05:54.664334 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:54.664141 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:05:54.916962 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:54.916503 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:54.916962 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:54.916586 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:55.664050 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:55.664008 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-lwsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 22:05:55.664240 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:55.664077 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 22:05:55.916312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:55.916219 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:55.916700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:55.916306 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:56.916082 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:56.916029 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-66hx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 22:05:56.916305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:56.916100 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 22:05:57.428736 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:57.428690 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:57.429911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:57.429866 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:57.430667 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:57.430647 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:05:57.920473 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:57.920445 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:57.920771 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:57.920748 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:05:57.921567 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:05:57.921548 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:06:03.052937 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.052899 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-4h2fw"] Oct 30 22:06:03.053453 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.053371 5914 cpu_manager.go:414] "containerMap: removing container" podUID="b59714cf-7528-4ece-a09e-3bad8db1bedb" containerName="manager" Oct 30 22:06:03.053453 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.053383 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59714cf-7528-4ece-a09e-3bad8db1bedb" containerName="manager" Oct 30 22:06:03.053453 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.053441 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59714cf-7528-4ece-a09e-3bad8db1bedb" containerName="manager" Oct 30 22:06:03.056647 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.056631 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.109137 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.109107 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 22:06:03.152957 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.152919 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-webhook-cert\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.153124 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.153097 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2nfk\" (UniqueName: \"kubernetes.io/projected/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-kube-api-access-k2nfk\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.153166 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.153135 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-metrics-certs\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.159272 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.159247 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 22:06:03.159544 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.159525 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ffhp5\"" Oct 30 22:06:03.159596 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.159525 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 22:06:03.161077 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.161005 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-4h2fw"] Oct 30 22:06:03.254012 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.253975 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2nfk\" (UniqueName: \"kubernetes.io/projected/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-kube-api-access-k2nfk\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.254012 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.254022 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-metrics-certs\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.254248 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.254166 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-webhook-cert\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.256725 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.256700 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-webhook-cert\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.256849 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.256825 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-metrics-certs\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.312139 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.312065 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2nfk\" (UniqueName: \"kubernetes.io/projected/3de5dbd8-6c4e-480c-9f2b-4236665c3dcc-kube-api-access-k2nfk\") pod \"maas-controller-58d6bcb5f-4h2fw\" (UID: \"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc\") " pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.369189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.369151 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:03.741078 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:03.741055 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58d6bcb5f-4h2fw"] Oct 30 22:06:03.743206 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:06:03.743179 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de5dbd8_6c4e_480c_9f2b_4236665c3dcc.slice/crio-1c37af6f73cf845eca3011c4dbeaef79a77dfc10aab8f39659f89d79f09b863a WatchSource:0}: Error finding container 1c37af6f73cf845eca3011c4dbeaef79a77dfc10aab8f39659f89d79f09b863a: Status 404 returned error can't find the container with id 1c37af6f73cf845eca3011c4dbeaef79a77dfc10aab8f39659f89d79f09b863a Oct 30 22:06:04.500616 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:04.500575 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" event={"ID":"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc","Type":"ContainerStarted","Data":"1c37af6f73cf845eca3011c4dbeaef79a77dfc10aab8f39659f89d79f09b863a"} Oct 30 22:06:05.507174 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:05.507133 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" event={"ID":"3de5dbd8-6c4e-480c-9f2b-4236665c3dcc","Type":"ContainerStarted","Data":"674200928a39d96c960b17abff18d39aea1706bca2385869f4f3d4287d991d8b"} Oct 30 22:06:05.507557 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:05.507210 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:05.640574 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:05.640525 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" podStartSLOduration=1.709302497 podStartE2EDuration="2.640511486s" totalImagesPullingTime="931.208989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:05.639322354 +0000 UTC m=+2917.541186411" watchObservedRunningTime="2026-10-30 22:06:05.640511486 +0000 UTC m=+2917.542375551" Oct 30 22:06:12.344625 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:06:12.344583 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:06:16.514919 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:16.514864 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58d6bcb5f-4h2fw" Oct 30 22:06:30.220626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:30.220592 5914 scope.go:122] "RemoveContainer" containerID="4b57f9ba738f4c8b1b0d23c5123fabfb83fe68278a2f048a82183f9bbbeac15c" Oct 30 22:06:42.345486 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:06:42.345445 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:06:53.957780 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:06:53.957726 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:07:12.345072 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:07:12.344991 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:07:15.646135 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:15.646098 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:07:18.384983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.384942 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw"] Oct 30 22:07:18.388737 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.388717 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.420811 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.420782 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421014 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.420820 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421014 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.420840 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421014 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.420931 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.421042 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.421076 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.421103 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27lcx\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.421159 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.421309 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.421206 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.521984 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522024 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522046 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27lcx\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522068 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522115 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522216 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522574 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522256 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522574 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522288 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522574 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522378 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522574 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522436 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522740 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522616 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522740 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522684 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.522740 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.522710 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.523152 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.523132 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.524598 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.524578 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.524850 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.524831 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.580014 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.579972 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27lcx\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.580351 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.580331 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mblkw\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:18.599460 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.599389 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw"] Oct 30 22:07:18.701788 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:18.701705 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:19.128756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:19.128729 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw"] Oct 30 22:07:19.131355 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:07:19.131328 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a11a6a_ccd2_414b_a170_d066c51576b1.slice/crio-2efd636207075dd59deef45984f832dad378a9c6e0ab9700591585308489bff3 WatchSource:0}: Error finding container 2efd636207075dd59deef45984f832dad378a9c6e0ab9700591585308489bff3: Status 404 returned error can't find the container with id 2efd636207075dd59deef45984f832dad378a9c6e0ab9700591585308489bff3 Oct 30 22:07:19.133454 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:19.133423 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.133536 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:19.133483 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.133536 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:19.133511 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:19.865231 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:19.865194 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" event={"ID":"f9a11a6a-ccd2-414b-a170-d066c51576b1","Type":"ContainerStarted","Data":"aa2f947c690309b80d978a1c1f685b64dcdf1dceddf6399ee2dc2768b6622fd9"} Oct 30 22:07:19.865756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:19.865243 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" event={"ID":"f9a11a6a-ccd2-414b-a170-d066c51576b1","Type":"ContainerStarted","Data":"2efd636207075dd59deef45984f832dad378a9c6e0ab9700591585308489bff3"} Oct 30 22:07:20.702348 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:20.702304 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:20.703976 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:20.703941 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mblkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:07:20.704131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:20.703996 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 22:07:21.702415 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:21.702373 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mblkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:07:21.702866 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:21.702433 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 22:07:22.702372 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:22.702337 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mblkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:07:22.702566 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:22.702391 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 22:07:23.702439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:23.702395 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mblkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:07:23.702644 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:23.702456 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 22:07:24.702235 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:24.702195 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mblkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:07:24.702459 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:24.702261 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 22:07:25.702765 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:25.702729 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mblkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 22:07:25.703191 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:25.702792 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 22:07:27.427527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:27.427492 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:27.428066 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:27.427914 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:27.428748 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:27.428726 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:07:27.593529 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:27.593473 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podStartSLOduration=9.593456007 podStartE2EDuration="9.593456007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:20.081752473 +0000 UTC m=+2991.983616540" watchObservedRunningTime="2026-10-30 22:07:27.593456007 +0000 UTC m=+2999.495320071" Oct 30 22:07:29.328479 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:29.328445 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:07:29.334418 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:29.334395 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:07:30.155698 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:30.155661 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:07:33.380213 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.380180 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx"] Oct 30 22:07:33.384070 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.384050 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.488359 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.488321 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx"] Oct 30 22:07:33.544500 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544465 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544500 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544506 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544776 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544537 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544776 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544703 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544776 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544761 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544987 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544829 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544987 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544913 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544987 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544933 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzkkp\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.544987 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.544956 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646440 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646352 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646440 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646389 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646660 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646513 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646660 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646598 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646660 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646634 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646787 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646678 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646787 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646735 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646787 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646762 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzkkp\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646803 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.646967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.646931 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.647098 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.647075 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.647252 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.647224 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.647321 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.647252 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.647321 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.647228 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.649323 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.649295 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.649541 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.649523 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.704465 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.704425 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.704465 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.704461 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzkkp\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp\") pod \"maas-default-gateway-openshift-default-7f694df8bc-zqcqx\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:33.997551 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:33.997465 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:34.420676 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:34.420632 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx"] Oct 30 22:07:34.423191 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:07:34.423161 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845ebdc3_490d_4413_bab1_6838ea3d3603.slice/crio-856ceecfaa34ffd37465c5cfdb6e6e7f5e2492f83a71423aa9605d9106bcccbf WatchSource:0}: Error finding container 856ceecfaa34ffd37465c5cfdb6e6e7f5e2492f83a71423aa9605d9106bcccbf: Status 404 returned error can't find the container with id 856ceecfaa34ffd37465c5cfdb6e6e7f5e2492f83a71423aa9605d9106bcccbf Oct 30 22:07:34.425442 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:34.425410 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:34.425537 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:34.425475 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:34.425537 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:34.425506 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:35.000562 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:35.000331 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" event={"ID":"845ebdc3-490d-4413-bab1-6838ea3d3603","Type":"ContainerStarted","Data":"9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8"} Oct 30 22:07:35.000562 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:35.000404 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" event={"ID":"845ebdc3-490d-4413-bab1-6838ea3d3603","Type":"ContainerStarted","Data":"856ceecfaa34ffd37465c5cfdb6e6e7f5e2492f83a71423aa9605d9106bcccbf"} Oct 30 22:07:35.116743 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:35.116688 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podStartSLOduration=2.116675087 podStartE2EDuration="2.116675087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:35.115447986 +0000 UTC m=+3007.017312092" watchObservedRunningTime="2026-10-30 22:07:35.116675087 +0000 UTC m=+3007.018539151" Oct 30 22:07:35.998067 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:35.998021 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:35.999713 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:35.999683 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zqcqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:07:35.999859 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:35.999743 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:07:36.998330 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:36.998295 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zqcqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:07:36.998761 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:36.998351 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:07:37.998644 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:37.998358 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zqcqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:07:37.998644 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:37.998425 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:07:38.998291 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:38.998252 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zqcqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:07:38.998451 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:38.998322 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:07:39.998290 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:39.998245 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zqcqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:07:39.998802 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:39.998310 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:07:40.998331 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:40.998296 5914 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-zqcqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 22:07:40.998724 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:40.998347 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 22:07:42.347004 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:07:42.346959 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:07:42.692561 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:42.692527 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:42.692900 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:42.692860 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:42.694039 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:42.694019 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:07:50.674105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:07:50.674065 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:08:07.657091 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:08:07.657057 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:08:12.345250 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:08:12.345205 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:08:38.364646 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:08:38.364560 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:08:42.346180 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:08:42.346136 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:08:48.957664 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:08:48.957621 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:09:01.455761 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:09:01.455721 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:09:12.344506 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:09:12.344464 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:09:23.971657 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:09:23.971614 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:09:33.272571 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:09:33.272524 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:09:42.344919 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:09:42.344859 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:09:52.050915 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:09:52.050864 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:10:12.344643 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:10:12.344557 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:10:17.768326 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:10:17.768295 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:10:38.455528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:10:38.455490 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:10:42.346116 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:10:42.346073 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:11:01.154375 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:11:01.154333 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:11:12.345041 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:11:12.345003 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:11:42.346024 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:11:42.345938 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:11:45.261916 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:11:45.261867 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:12:12.345000 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:12:12.344960 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:12:28.489622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:12:28.489585 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:12:29.383844 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:12:29.383804 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:12:29.388491 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:12:29.388462 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:12:39.152820 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:12:39.152780 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:12:42.345900 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:12:42.345841 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:13:12.345014 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:13:12.344924 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:13:15.873954 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:15.873918 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6"] Oct 30 22:13:15.877413 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:15.877391 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:15.929973 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:15.929941 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c3e01b-openshift-default-dockercfg-76hx7\"" Oct 30 22:13:15.936404 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:15.936371 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6"] Oct 30 22:13:16.029944 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.029909 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.029953 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.029991 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.030027 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.030084 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhl2g\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.030113 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030422 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.030190 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030422 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.030266 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.030422 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.030296 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.131729 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131628 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.131729 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131666 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.131729 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131690 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.131729 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131713 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132114 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131742 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhl2g\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132114 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131764 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132114 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.131819 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132293 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.132108 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132293 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.132222 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132293 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.132255 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132633 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.132606 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132724 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.132710 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.132828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.132807 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.134562 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.134524 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.134842 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.134816 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.135074 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.135057 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.192340 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.192309 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.192484 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.192363 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhl2g\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g\") pod \"e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.489942 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.489818 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:16.930470 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.930433 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6"] Oct 30 22:13:16.933271 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:13:16.933236 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0daa88_e0ff_45d2_a825_afac33a60efc.slice/crio-7d5b614a5151e1f9535b653710ee6523b81b57437feb1d1a98a8ccfea17fe287 WatchSource:0}: Error finding container 7d5b614a5151e1f9535b653710ee6523b81b57437feb1d1a98a8ccfea17fe287: Status 404 returned error can't find the container with id 7d5b614a5151e1f9535b653710ee6523b81b57437feb1d1a98a8ccfea17fe287 Oct 30 22:13:16.935538 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.935506 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:16.935620 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.935570 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:16.935620 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:16.935600 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:17.850377 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:17.850336 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" event={"ID":"ed0daa88-e0ff-45d2-a825-afac33a60efc","Type":"ContainerStarted","Data":"025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de"} Oct 30 22:13:17.850605 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:17.850399 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" event={"ID":"ed0daa88-e0ff-45d2-a825-afac33a60efc","Type":"ContainerStarted","Data":"7d5b614a5151e1f9535b653710ee6523b81b57437feb1d1a98a8ccfea17fe287"} Oct 30 22:13:18.018827 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:18.018747 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podStartSLOduration=3.018728701 podStartE2EDuration="3.018728701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:18.018460699 +0000 UTC m=+3349.920324766" watchObservedRunningTime="2026-10-30 22:13:18.018728701 +0000 UTC m=+3349.920592765" Oct 30 22:13:18.490795 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:18.490762 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:18.496233 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:18.496206 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:18.855828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:18.855741 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:18.856859 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:18.856840 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:24.748246 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.748210 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr"] Oct 30 22:13:24.751834 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.751817 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:24.806150 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.806118 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c3e01b\"" Oct 30 22:13:24.818956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.818921 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7l5n\" (UniqueName: \"kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n\") pod \"payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:24.818956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.818961 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:24.855987 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.855959 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a4842f85-dockercfg-tkqh4\"" Oct 30 22:13:24.857641 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.857612 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr"] Oct 30 22:13:24.918077 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.918042 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt"] Oct 30 22:13:24.919975 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.919947 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7l5n\" (UniqueName: \"kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n\") pod \"payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:24.919975 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.919979 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:24.920665 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.920646 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:24.921865 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.921846 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:24.978843 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.978746 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt"] Oct 30 22:13:24.980578 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:24.980553 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7l5n\" (UniqueName: \"kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n\") pod \"payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:25.020995 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.020908 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r48\" (UniqueName: \"kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48\") pod \"payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.020995 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.020971 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.066155 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.066125 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:25.121737 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.121666 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.121900 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.121812 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r48\" (UniqueName: \"kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48\") pod \"payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.127329 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.122633 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.179592 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.179559 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r48\" (UniqueName: \"kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48\") pod \"payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.235158 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.235121 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.440749 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.440719 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr"] Oct 30 22:13:25.443071 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:13:25.443046 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ded3f4_10e4_4ade_bfcd_bc0f274b20a0.slice/crio-6d13c2d41be09ff5371c9cc30eb44b18ab1b0c465abd715dd577968e1bfbee11 WatchSource:0}: Error finding container 6d13c2d41be09ff5371c9cc30eb44b18ab1b0c465abd715dd577968e1bfbee11: Status 404 returned error can't find the container with id 6d13c2d41be09ff5371c9cc30eb44b18ab1b0c465abd715dd577968e1bfbee11 Oct 30 22:13:25.609407 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.609382 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt"] Oct 30 22:13:25.611635 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:13:25.611608 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c2b57a_7602_40fa_886b_94da164f00dc.slice/crio-921be25ac901d3b646676709f75720f4cbe1e13113ed4db1be5da1a880333722 WatchSource:0}: Error finding container 921be25ac901d3b646676709f75720f4cbe1e13113ed4db1be5da1a880333722: Status 404 returned error can't find the container with id 921be25ac901d3b646676709f75720f4cbe1e13113ed4db1be5da1a880333722 Oct 30 22:13:25.891717 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.891675 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" event={"ID":"62c2b57a-7602-40fa-886b-94da164f00dc","Type":"ContainerStarted","Data":"25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15"} Oct 30 22:13:25.892189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.891774 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" event={"ID":"62c2b57a-7602-40fa-886b-94da164f00dc","Type":"ContainerStarted","Data":"921be25ac901d3b646676709f75720f4cbe1e13113ed4db1be5da1a880333722"} Oct 30 22:13:25.892189 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.891810 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:25.893320 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.893293 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" event={"ID":"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0","Type":"ContainerStarted","Data":"40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f"} Oct 30 22:13:25.893460 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.893344 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" event={"ID":"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0","Type":"ContainerStarted","Data":"6d13c2d41be09ff5371c9cc30eb44b18ab1b0c465abd715dd577968e1bfbee11"} Oct 30 22:13:25.893460 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:25.893372 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:26.007929 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:26.007845 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" podStartSLOduration=2.0078324 podStartE2EDuration="2.0078324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:26.006357392 +0000 UTC m=+3357.908221457" watchObservedRunningTime="2026-10-30 22:13:26.0078324 +0000 UTC m=+3357.909696459" Oct 30 22:13:26.123053 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:26.122444 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" podStartSLOduration=2.122429836 podStartE2EDuration="2.122429836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:26.120598484 +0000 UTC m=+3358.022462550" watchObservedRunningTime="2026-10-30 22:13:26.122429836 +0000 UTC m=+3358.024293902" Oct 30 22:13:27.142826 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.142787 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8"] Oct 30 22:13:27.146499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.146483 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.201804 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.201766 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c3e01b-serving-cert\"" Oct 30 22:13:27.204100 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.204074 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8"] Oct 30 22:13:27.243629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.243591 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkhb\" (UniqueName: \"kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.243801 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.243642 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.243801 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.243667 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.243801 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.243690 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.344615 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.344576 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkhb\" (UniqueName: \"kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.344790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.344628 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.344790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.344649 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.344790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.344675 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.344940 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:13:27.344789 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-serving-cert: secret "maas-api-e2e-worker-single-main-c3e01b-serving-cert" not found Oct 30 22:13:27.344940 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:13:27.344868 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls podName:38f32ad4-d105-412a-94ef-622fddbc2aaa nodeName:}" failed. No retries permitted until 2026-10-30 22:13:27.844845687 +0000 UTC m=+3359.746709747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls") pod "maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" (UID: "38f32ad4-d105-412a-94ef-622fddbc2aaa") : secret "maas-api-e2e-worker-single-main-c3e01b-serving-cert" not found Oct 30 22:13:27.345449 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.345414 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.347378 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.347356 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.403018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.402941 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkhb\" (UniqueName: \"kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.851181 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.851095 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:27.853751 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:27.853728 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:28.060116 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:28.060079 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:28.437561 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:28.437532 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8"] Oct 30 22:13:28.439815 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:13:28.439782 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f32ad4_d105_412a_94ef_622fddbc2aaa.slice/crio-038bf9cc8ca7dfb0cd6d64b6831104acd006260d1cacdae1ddf2532c83302de4 WatchSource:0}: Error finding container 038bf9cc8ca7dfb0cd6d64b6831104acd006260d1cacdae1ddf2532c83302de4: Status 404 returned error can't find the container with id 038bf9cc8ca7dfb0cd6d64b6831104acd006260d1cacdae1ddf2532c83302de4 Oct 30 22:13:28.441453 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:28.441418 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:13:28.908622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:28.908585 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" event={"ID":"38f32ad4-d105-412a-94ef-622fddbc2aaa","Type":"ContainerStarted","Data":"038bf9cc8ca7dfb0cd6d64b6831104acd006260d1cacdae1ddf2532c83302de4"} Oct 30 22:13:29.915851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:29.915808 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" event={"ID":"38f32ad4-d105-412a-94ef-622fddbc2aaa","Type":"ContainerStarted","Data":"e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde"} Oct 30 22:13:29.916327 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:29.916000 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:30.031819 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:30.031754 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" podStartSLOduration=2.137202084 podStartE2EDuration="3.031733027s" totalImagesPullingTime="894.530943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:30.029867971 +0000 UTC m=+3361.931732036" watchObservedRunningTime="2026-10-30 22:13:30.031733027 +0000 UTC m=+3361.933597092" Oct 30 22:13:35.186829 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:35.186782 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:13:35.926656 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:35.926625 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:13:36.900738 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:36.900705 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:13:36.901193 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:36.900775 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:13:42.345508 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:13:42.345462 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:13:44.994173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:44.994142 5914 generic.go:356] "Generic (PLEG): container finished" podID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerID="025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de" exitCode=0 Oct 30 22:13:44.994650 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:44.994211 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" event={"ID":"ed0daa88-e0ff-45d2-a825-afac33a60efc","Type":"ContainerDied","Data":"025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de"} Oct 30 22:13:44.994650 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:44.994603 5914 scope.go:122] "RemoveContainer" containerID="025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de" Oct 30 22:13:44.995958 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:44.995935 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:44.996029 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:44.996000 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:44.996070 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:44.996032 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:13:46.003736 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:46.003704 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" event={"ID":"ed0daa88-e0ff-45d2-a825-afac33a60efc","Type":"ContainerStarted","Data":"773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f"} Oct 30 22:13:46.495272 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:46.495236 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:46.497217 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:46.497185 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:13:46.497516 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:46.497486 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:13:47.498062 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:47.498025 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:13:47.498601 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:47.498095 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:13:48.498094 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:48.498037 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:13:48.498588 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:48.498104 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:13:48.857097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:48.857000 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:49.342895 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.342852 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48"] Oct 30 22:13:49.346654 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.346638 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.426758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.426714 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9sp\" (UniqueName: \"kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.426973 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.426777 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.426973 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.426937 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.427108 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.427075 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.427170 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.427121 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.427170 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.427150 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.438875 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.438846 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c3e01b\"/\"openshift-service-ca.crt\"" Oct 30 22:13:49.449129 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.449094 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c3e01b\"/\"simulator-single-main-c3e01b-kserve-self-signed-certs\"" Oct 30 22:13:49.449129 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.449122 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c3e01b\"/\"kube-root-ca.crt\"" Oct 30 22:13:49.449315 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.449191 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c3e01b\"/\"default-dockercfg-zrfc8\"" Oct 30 22:13:49.450753 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.450730 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48"] Oct 30 22:13:49.493903 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.491290 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:13:49.493903 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.491354 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:13:49.528044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528011 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528086 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528106 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528120 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528170 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9sp\" (UniqueName: \"kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528201 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528473 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528509 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.528533 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.528537 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.530704 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.530680 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.530978 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.530961 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.599438 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.599350 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9sp\" (UniqueName: \"kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp\") pod \"simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:49.672648 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:49.672617 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:50.087832 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:50.074991 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48"] Oct 30 22:13:50.099218 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:13:50.099178 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c10b140_2711_472d_9350_6f65fa15099e.slice/crio-d71b626f9c95a223fede74ae02fa67ec2f33f141312ed235c273c3558eabff5c WatchSource:0}: Error finding container d71b626f9c95a223fede74ae02fa67ec2f33f141312ed235c273c3558eabff5c: Status 404 returned error can't find the container with id d71b626f9c95a223fede74ae02fa67ec2f33f141312ed235c273c3558eabff5c Oct 30 22:13:50.497906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:50.495244 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 22:13:50.497906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:50.495306 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 22:13:51.033687 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:51.033652 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" event={"ID":"2c10b140-2711-472d-9350-6f65fa15099e","Type":"ContainerStarted","Data":"05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff"} Oct 30 22:13:51.034228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:51.033704 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" event={"ID":"2c10b140-2711-472d-9350-6f65fa15099e","Type":"ContainerStarted","Data":"d71b626f9c95a223fede74ae02fa67ec2f33f141312ed235c273c3558eabff5c"} Oct 30 22:13:51.150758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:51.150704 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" podStartSLOduration=2.150688642 podStartE2EDuration="2.150688642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:51.148774432 +0000 UTC m=+3383.050638510" watchObservedRunningTime="2026-10-30 22:13:51.150688642 +0000 UTC m=+3383.052552707" Oct 30 22:13:51.495017 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:51.494982 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:51.495977 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:51.495954 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:13:51.673136 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:51.673099 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:54.149655 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:54.149620 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:13:59.673417 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:59.673375 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:13:59.686198 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:13:59.686171 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:14:00.090261 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:14:00.090170 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:14:12.345645 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:14:12.345602 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:14:29.778546 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:14:29.778509 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:14:42.345319 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:14:42.345227 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:15:09.197432 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:09.197386 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:15:11.022173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:11.022138 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48"] Oct 30 22:15:12.345148 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:12.345110 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:15:18.059076 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:18.059039 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr"] Oct 30 22:15:18.168801 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:18.168764 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt"] Oct 30 22:15:23.059818 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:23.059772 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerName="payload-processing" containerID="cri-o://40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f" gracePeriod=25 Oct 30 22:15:23.169065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:23.169019 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" containerName="payload-pre-processing" containerID="cri-o://25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15" gracePeriod=25 Oct 30 22:15:23.911629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:23.911586 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8"] Oct 30 22:15:23.911825 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:23.911794 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" podUID="38f32ad4-d105-412a-94ef-622fddbc2aaa" containerName="maas-api" containerID="cri-o://e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde" gracePeriod=30 Oct 30 22:15:25.167958 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.167928 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:15:25.248103 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.248008 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb\" (UniqueName: \"kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb\") pod \"38f32ad4-d105-412a-94ef-622fddbc2aaa\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " Oct 30 22:15:25.248284 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.248217 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs\") pod \"38f32ad4-d105-412a-94ef-622fddbc2aaa\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " Oct 30 22:15:25.248284 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.248254 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\") pod \"38f32ad4-d105-412a-94ef-622fddbc2aaa\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " Oct 30 22:15:25.248409 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.248293 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca\") pod \"38f32ad4-d105-412a-94ef-622fddbc2aaa\" (UID: \"38f32ad4-d105-412a-94ef-622fddbc2aaa\") " Oct 30 22:15:25.248830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.248768 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca" pod "38f32ad4-d105-412a-94ef-622fddbc2aaa" (UID: "38f32ad4-d105-412a-94ef-622fddbc2aaa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:25.250339 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.250315 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb" pod "38f32ad4-d105-412a-94ef-622fddbc2aaa" (UID: "38f32ad4-d105-412a-94ef-622fddbc2aaa"). InnerVolumeSpecName "kube-api-access-zgkhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:25.250455 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.250439 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs" pod "38f32ad4-d105-412a-94ef-622fddbc2aaa" (UID: "38f32ad4-d105-412a-94ef-622fddbc2aaa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:25.250561 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.250544 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls" pod "38f32ad4-d105-412a-94ef-622fddbc2aaa" (UID: "38f32ad4-d105-412a-94ef-622fddbc2aaa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:25.349689 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.349653 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkhb\" (UniqueName: \"kubernetes.io/projected/38f32ad4-d105-412a-94ef-622fddbc2aaa-kube-api-access-zgkhb\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.349689 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.349683 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.349689 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.349694 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f32ad4-d105-412a-94ef-622fddbc2aaa-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.349973 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.349702 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f32ad4-d105-412a-94ef-622fddbc2aaa-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:25.513085 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.512996 5914 generic.go:356] "Generic (PLEG): container finished" podID="38f32ad4-d105-412a-94ef-622fddbc2aaa" containerID="e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde" exitCode=0 Oct 30 22:15:25.513085 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.513043 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" Oct 30 22:15:25.513085 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.513052 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" event={"ID":"38f32ad4-d105-412a-94ef-622fddbc2aaa","Type":"ContainerDied","Data":"e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde"} Oct 30 22:15:25.513373 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.513101 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8" event={"ID":"38f32ad4-d105-412a-94ef-622fddbc2aaa","Type":"ContainerDied","Data":"038bf9cc8ca7dfb0cd6d64b6831104acd006260d1cacdae1ddf2532c83302de4"} Oct 30 22:15:25.513373 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.513124 5914 scope.go:122] "RemoveContainer" containerID="e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde" Oct 30 22:15:25.523103 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.523083 5914 scope.go:122] "RemoveContainer" containerID="e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde" Oct 30 22:15:25.523402 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:25.523387 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde\": container with ID starting with e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde not found: ID does not exist" containerID="e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde" Oct 30 22:15:25.523456 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.523412 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde"} err="failed to get container status \"e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde\": rpc error: code = NotFound desc = could not find container \"e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde\": container with ID starting with e34a6cfcd0681ee63a98949fda900333ed8349858374973f7fa28012aee48fde not found: ID does not exist" Oct 30 22:15:25.675606 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.675568 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8"] Oct 30 22:15:25.679855 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:25.679828 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c3e01b-5888587ddb-vg5f8"] Oct 30 22:15:26.046362 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.046322 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" podUID="2c10b140-2711-472d-9350-6f65fa15099e" containerName="main" containerID="cri-o://05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff" gracePeriod=45 Oct 30 22:15:26.080222 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.080189 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" podUID="2c10b140-2711-472d-9350-6f65fa15099e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.150:8000/ready\": dial tcp 10.134.0.150:8000: connect: connection refused" Oct 30 22:15:26.217219 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.217180 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f32ad4-d105-412a-94ef-622fddbc2aaa" path="/var/lib/kubelet/pods/38f32ad4-d105-412a-94ef-622fddbc2aaa/volumes" Oct 30 22:15:26.306633 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.306569 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:15:26.359929 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.359897 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home\") pod \"2c10b140-2711-472d-9350-6f65fa15099e\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " Oct 30 22:15:26.360125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.359956 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp\" (UniqueName: \"kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp\") pod \"2c10b140-2711-472d-9350-6f65fa15099e\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " Oct 30 22:15:26.360125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360015 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm\") pod \"2c10b140-2711-472d-9350-6f65fa15099e\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " Oct 30 22:15:26.360234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360151 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs\") pod \"2c10b140-2711-472d-9350-6f65fa15099e\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " Oct 30 22:15:26.360234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360171 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home" pod "2c10b140-2711-472d-9350-6f65fa15099e" (UID: "2c10b140-2711-472d-9350-6f65fa15099e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.360234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360192 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir\") pod \"2c10b140-2711-472d-9350-6f65fa15099e\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " Oct 30 22:15:26.360387 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360257 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache\") pod \"2c10b140-2711-472d-9350-6f65fa15099e\" (UID: \"2c10b140-2711-472d-9350-6f65fa15099e\") " Oct 30 22:15:26.360514 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360484 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir" pod "2c10b140-2711-472d-9350-6f65fa15099e" (UID: "2c10b140-2711-472d-9350-6f65fa15099e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.360588 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360537 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache" pod "2c10b140-2711-472d-9350-6f65fa15099e" (UID: "2c10b140-2711-472d-9350-6f65fa15099e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.360665 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360649 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.360725 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360673 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.360725 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.360690 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.362543 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.362520 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs" pod "2c10b140-2711-472d-9350-6f65fa15099e" (UID: "2c10b140-2711-472d-9350-6f65fa15099e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:26.362665 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.362618 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp" pod "2c10b140-2711-472d-9350-6f65fa15099e" (UID: "2c10b140-2711-472d-9350-6f65fa15099e"). InnerVolumeSpecName "kube-api-access-mh9sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:26.363160 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.363140 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm" pod "2c10b140-2711-472d-9350-6f65fa15099e" (UID: "2c10b140-2711-472d-9350-6f65fa15099e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:26.461602 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.461564 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh9sp\" (UniqueName: \"kubernetes.io/projected/2c10b140-2711-472d-9350-6f65fa15099e-kube-api-access-mh9sp\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.461602 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.461597 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c10b140-2711-472d-9350-6f65fa15099e-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.461602 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.461607 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c10b140-2711-472d-9350-6f65fa15099e-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:26.519831 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.519798 5914 generic.go:356] "Generic (PLEG): container finished" podID="2c10b140-2711-472d-9350-6f65fa15099e" containerID="05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff" exitCode=0 Oct 30 22:15:26.520068 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.519847 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" Oct 30 22:15:26.520068 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.519920 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" event={"ID":"2c10b140-2711-472d-9350-6f65fa15099e","Type":"ContainerDied","Data":"05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff"} Oct 30 22:15:26.520068 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.519967 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48" event={"ID":"2c10b140-2711-472d-9350-6f65fa15099e","Type":"ContainerDied","Data":"d71b626f9c95a223fede74ae02fa67ec2f33f141312ed235c273c3558eabff5c"} Oct 30 22:15:26.520068 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.519990 5914 scope.go:122] "RemoveContainer" containerID="05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff" Oct 30 22:15:26.529723 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.529704 5914 scope.go:122] "RemoveContainer" containerID="05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff" Oct 30 22:15:26.530018 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:26.530001 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff\": container with ID starting with 05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff not found: ID does not exist" containerID="05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff" Oct 30 22:15:26.530098 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.530026 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff"} err="failed to get container status \"05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff\": rpc error: code = NotFound desc = could not find container \"05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff\": container with ID starting with 05878260f497d1516d5393c66078c6e88c48a0b3cc759f60e5aa6adef18ce0ff not found: ID does not exist" Oct 30 22:15:26.685516 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.685480 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48"] Oct 30 22:15:26.694121 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.694091 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c3e01b/simulator-single-main-c3e01b-kserve-5bf4c9b5bb-jgx48"] Oct 30 22:15:26.899765 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.899726 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:26.899963 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:26.899738 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:28.217813 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:28.217783 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c10b140-2711-472d-9350-6f65fa15099e" path="/var/lib/kubelet/pods/2c10b140-2711-472d-9350-6f65fa15099e/volumes" Oct 30 22:15:28.690519 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:28.690486 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:15:32.927814 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:32.927776 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6"] Oct 30 22:15:32.928325 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:32.928010 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" containerID="cri-o://773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f" gracePeriod=30 Oct 30 22:15:36.497988 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:36.497942 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:15:36.498389 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:36.498029 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:15:36.899353 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:36.899315 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:36.899353 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:36.899334 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:38.191837 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.191778 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:15:38.284593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284540 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284666 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284690 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284770 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284763 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284876 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284816 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:38.284876 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284831 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284995 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284932 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284995 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284964 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.284995 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.284985 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.285149 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285009 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo\") pod \"ed0daa88-e0ff-45d2-a825-afac33a60efc\" (UID: \"ed0daa88-e0ff-45d2-a825-afac33a60efc\") " Oct 30 22:15:38.285211 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285139 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:38.285329 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285296 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:38.285329 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285304 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:38.285563 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285434 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:38.285563 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285556 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.285796 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285575 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.285796 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285590 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed0daa88-e0ff-45d2-a825-afac33a60efc-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.285796 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285603 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.285796 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.285617 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.287099 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.287075 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:38.287308 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.287290 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:38.287760 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.287734 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:15:38.287760 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.287741 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g" pod "ed0daa88-e0ff-45d2-a825-afac33a60efc" (UID: "ed0daa88-e0ff-45d2-a825-afac33a60efc"). InnerVolumeSpecName "kube-api-access-xhl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:38.386185 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.386155 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.386185 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.386184 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhl2g\" (UniqueName: \"kubernetes.io/projected/ed0daa88-e0ff-45d2-a825-afac33a60efc-kube-api-access-xhl2g\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.386382 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.386194 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.386382 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.386204 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed0daa88-e0ff-45d2-a825-afac33a60efc-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:38.584643 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.584547 5914 generic.go:356] "Generic (PLEG): container finished" podID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerID="773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f" exitCode=0 Oct 30 22:15:38.584643 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.584602 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" Oct 30 22:15:38.584643 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.584631 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" event={"ID":"ed0daa88-e0ff-45d2-a825-afac33a60efc","Type":"ContainerDied","Data":"773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f"} Oct 30 22:15:38.584943 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.584685 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6" event={"ID":"ed0daa88-e0ff-45d2-a825-afac33a60efc","Type":"ContainerDied","Data":"7d5b614a5151e1f9535b653710ee6523b81b57437feb1d1a98a8ccfea17fe287"} Oct 30 22:15:38.584943 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.584709 5914 scope.go:122] "RemoveContainer" containerID="773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f" Oct 30 22:15:38.597984 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.597956 5914 scope.go:122] "RemoveContainer" containerID="025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de" Oct 30 22:15:38.608373 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.608354 5914 scope.go:122] "RemoveContainer" containerID="773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f" Oct 30 22:15:38.608672 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:38.608653 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f\": container with ID starting with 773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f not found: ID does not exist" containerID="773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f" Oct 30 22:15:38.608766 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.608685 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f"} err="failed to get container status \"773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f\": rpc error: code = NotFound desc = could not find container \"773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f\": container with ID starting with 773c527b08da3ceb6acef8671b951a27e755287e0313cfc999e95de88e605c7f not found: ID does not exist" Oct 30 22:15:38.608766 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.608710 5914 scope.go:122] "RemoveContainer" containerID="025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de" Oct 30 22:15:38.608975 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:38.608949 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de\": container with ID starting with 025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de not found: ID does not exist" containerID="025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de" Oct 30 22:15:38.609021 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.608987 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de"} err="failed to get container status \"025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de\": rpc error: code = NotFound desc = could not find container \"025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de\": container with ID starting with 025139084fc1d1fc2fa884db82c988467203a5741e7e28c716f1e7a4f76915de not found: ID does not exist" Oct 30 22:15:38.747911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.747863 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6"] Oct 30 22:15:38.751385 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:38.751360 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c3e01b-openshift-default-6984d596f-xjsl6"] Oct 30 22:15:40.216028 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:40.215997 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" path="/var/lib/kubelet/pods/ed0daa88-e0ff-45d2-a825-afac33a60efc/volumes" Oct 30 22:15:42.345797 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:42.345761 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:15:43.313019 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.312990 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:15:43.422065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.422043 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:15:43.437516 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.437484 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume\") pod \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " Oct 30 22:15:43.437679 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.437601 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n\" (UniqueName: \"kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n\") pod \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\" (UID: \"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0\") " Oct 30 22:15:43.437929 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.437854 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume" pod "44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" (UID: "44ded3f4-10e4-4ade-bfcd-bc0f274b20a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:43.440121 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.440096 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n" pod "44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" (UID: "44ded3f4-10e4-4ade-bfcd-bc0f274b20a0"). InnerVolumeSpecName "kube-api-access-g7l5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:43.539182 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.539146 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume\") pod \"62c2b57a-7602-40fa-886b-94da164f00dc\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " Oct 30 22:15:43.539334 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.539262 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48\" (UniqueName: \"kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48\") pod \"62c2b57a-7602-40fa-886b-94da164f00dc\" (UID: \"62c2b57a-7602-40fa-886b-94da164f00dc\") " Oct 30 22:15:43.539612 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.539590 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume" pod "62c2b57a-7602-40fa-886b-94da164f00dc" (UID: "62c2b57a-7602-40fa-886b-94da164f00dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:43.539673 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.539660 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7l5n\" (UniqueName: \"kubernetes.io/projected/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-kube-api-access-g7l5n\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:43.539716 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.539677 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:43.541546 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.541523 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48" pod "62c2b57a-7602-40fa-886b-94da164f00dc" (UID: "62c2b57a-7602-40fa-886b-94da164f00dc"). InnerVolumeSpecName "kube-api-access-k9r48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:43.611575 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.611494 5914 generic.go:356] "Generic (PLEG): container finished" podID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerID="40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f" exitCode=0 Oct 30 22:15:43.611575 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.611538 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" Oct 30 22:15:43.611794 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.611577 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" event={"ID":"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0","Type":"ContainerDied","Data":"40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f"} Oct 30 22:15:43.611794 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.611633 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr" event={"ID":"44ded3f4-10e4-4ade-bfcd-bc0f274b20a0","Type":"ContainerDied","Data":"6d13c2d41be09ff5371c9cc30eb44b18ab1b0c465abd715dd577968e1bfbee11"} Oct 30 22:15:43.611794 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.611661 5914 scope.go:122] "RemoveContainer" containerID="40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f" Oct 30 22:15:43.613104 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.613078 5914 generic.go:356] "Generic (PLEG): container finished" podID="62c2b57a-7602-40fa-886b-94da164f00dc" containerID="25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15" exitCode=0 Oct 30 22:15:43.613231 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.613124 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" Oct 30 22:15:43.613231 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.613197 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" event={"ID":"62c2b57a-7602-40fa-886b-94da164f00dc","Type":"ContainerDied","Data":"25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15"} Oct 30 22:15:43.613325 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.613232 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt" event={"ID":"62c2b57a-7602-40fa-886b-94da164f00dc","Type":"ContainerDied","Data":"921be25ac901d3b646676709f75720f4cbe1e13113ed4db1be5da1a880333722"} Oct 30 22:15:43.620947 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.620804 5914 scope.go:122] "RemoveContainer" containerID="40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f" Oct 30 22:15:43.621117 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:43.621095 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f\": container with ID starting with 40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f not found: ID does not exist" containerID="40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f" Oct 30 22:15:43.621204 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.621128 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f"} err="failed to get container status \"40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f\": rpc error: code = NotFound desc = could not find container \"40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f\": container with ID starting with 40db8cb37da0655b01379f1e722fb654c487944a60538b0fe90aba648ce7fe2f not found: ID does not exist" Oct 30 22:15:43.621204 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.621151 5914 scope.go:122] "RemoveContainer" containerID="25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15" Oct 30 22:15:43.629269 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.629248 5914 scope.go:122] "RemoveContainer" containerID="25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15" Oct 30 22:15:43.629539 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:15:43.629518 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15\": container with ID starting with 25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15 not found: ID does not exist" containerID="25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15" Oct 30 22:15:43.629599 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.629547 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15"} err="failed to get container status \"25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15\": rpc error: code = NotFound desc = could not find container \"25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15\": container with ID starting with 25d871724a1b4082181ee5d0171078630ee4955438c5ff7c0f4051b28eb56c15 not found: ID does not exist" Oct 30 22:15:43.640611 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.640586 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c2b57a-7602-40fa-886b-94da164f00dc-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:43.640708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.640615 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9r48\" (UniqueName: \"kubernetes.io/projected/62c2b57a-7602-40fa-886b-94da164f00dc-kube-api-access-k9r48\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:43.774099 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.774066 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt"] Oct 30 22:15:43.781808 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.781778 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c3e01b-6775fd25nt"] Oct 30 22:15:43.896057 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.896022 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr"] Oct 30 22:15:43.901105 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:43.901073 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c3e01b-7ffd4bf74lbkmr"] Oct 30 22:15:44.215835 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:44.215753 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" path="/var/lib/kubelet/pods/44ded3f4-10e4-4ade-bfcd-bc0f274b20a0/volumes" Oct 30 22:15:44.216175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:15:44.216162 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" path="/var/lib/kubelet/pods/62c2b57a-7602-40fa-886b-94da164f00dc/volumes" Oct 30 22:16:12.344925 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:16:12.344823 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:16:42.345102 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:16:42.345062 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:17:00.324516 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324481 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt"] Oct 30 22:17:00.324934 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324925 5914 cpu_manager.go:414] "containerMap: removing container" podUID="38f32ad4-d105-412a-94ef-622fddbc2aaa" containerName="maas-api" Oct 30 22:17:00.324983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324938 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f32ad4-d105-412a-94ef-622fddbc2aaa" containerName="maas-api" Oct 30 22:17:00.324983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324953 5914 cpu_manager.go:414] "containerMap: removing container" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" containerName="payload-pre-processing" Oct 30 22:17:00.324983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324958 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" containerName="payload-pre-processing" Oct 30 22:17:00.324983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324968 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" Oct 30 22:17:00.324983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324973 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" Oct 30 22:17:00.324983 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324983 5914 cpu_manager.go:414] "containerMap: removing container" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerName="payload-processing" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.324989 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerName="payload-processing" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325013 5914 cpu_manager.go:414] "containerMap: removing container" podUID="2c10b140-2711-472d-9350-6f65fa15099e" containerName="main" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325017 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c10b140-2711-472d-9350-6f65fa15099e" containerName="main" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325027 5914 cpu_manager.go:414] "containerMap: removing container" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325031 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325090 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c2b57a-7602-40fa-886b-94da164f00dc" containerName="payload-pre-processing" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325099 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325106 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ded3f4-10e4-4ade-bfcd-bc0f274b20a0" containerName="payload-processing" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325115 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c10b140-2711-472d-9350-6f65fa15099e" containerName="main" Oct 30 22:17:00.325173 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.325123 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f32ad4-d105-412a-94ef-622fddbc2aaa" containerName="maas-api" Oct 30 22:17:00.328131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.328110 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.379541 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.379503 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt"] Oct 30 22:17:00.414949 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.414911 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzt2g\" (UniqueName: \"kubernetes.io/projected/55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b-kube-api-access-dzt2g\") pod \"kuadrant-operator-controller-manager-678d87766f-4xdrt\" (UID: \"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.415109 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.415023 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4xdrt\" (UID: \"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.427343 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.427308 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cxzr5\"" Oct 30 22:17:00.515905 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.515845 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzt2g\" (UniqueName: \"kubernetes.io/projected/55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b-kube-api-access-dzt2g\") pod \"kuadrant-operator-controller-manager-678d87766f-4xdrt\" (UID: \"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.516090 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.515986 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4xdrt\" (UID: \"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.516344 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.516327 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4xdrt\" (UID: \"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.573071 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.573036 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzt2g\" (UniqueName: \"kubernetes.io/projected/55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b-kube-api-access-dzt2g\") pod \"kuadrant-operator-controller-manager-678d87766f-4xdrt\" (UID: \"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:00.641347 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:00.641320 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:01.118242 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:01.118218 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt"] Oct 30 22:17:01.120786 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:17:01.120748 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c17f4d_2ea4_4bf3_a9a2_1e2859f9288b.slice/crio-0a231a8e0b757e81ac7e58a65df23b5feded0d1230060f3d795b503e07cf58f4 WatchSource:0}: Error finding container 0a231a8e0b757e81ac7e58a65df23b5feded0d1230060f3d795b503e07cf58f4: Status 404 returned error can't find the container with id 0a231a8e0b757e81ac7e58a65df23b5feded0d1230060f3d795b503e07cf58f4 Oct 30 22:17:01.999768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:01.999734 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" event={"ID":"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b","Type":"ContainerStarted","Data":"0a231a8e0b757e81ac7e58a65df23b5feded0d1230060f3d795b503e07cf58f4"} Oct 30 22:17:08.034649 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:08.034614 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" event={"ID":"55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b","Type":"ContainerStarted","Data":"f5f70110f6bb0d1dced31c244c925cd1d86ce7f641e3acc49c163620ce61af32"} Oct 30 22:17:08.035078 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:08.034675 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:08.251396 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:08.251334 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" podStartSLOduration=2.1479065569999998 podStartE2EDuration="8.251319468s" totalImagesPullingTime="6.103412911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:08.250247005 +0000 UTC m=+3580.152111073" watchObservedRunningTime="2026-10-30 22:17:08.251319468 +0000 UTC m=+3580.153183533" Oct 30 22:17:10.249068 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:10.249025 5914 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal" podUID="d6458e4c6fa120c378f995c71dc796f0" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:17:12.344978 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:17:12.344938 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:17:19.042142 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:19.042105 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4xdrt" Oct 30 22:17:28.170932 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:28.170894 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:17:29.433541 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:29.433505 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:17:29.441290 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:29.441260 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:17:42.345629 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:17:42.345542 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:17:48.589995 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:48.589957 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:17:57.909573 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:17:57.909537 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:18:12.345795 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:18:12.345757 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:18:27.697177 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:27.697140 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:18:42.344930 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:18:42.344874 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:18:47.860527 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.860491 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb"] Oct 30 22:18:47.863053 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.861072 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0daa88-e0ff-45d2-a825-afac33a60efc" containerName="istio-proxy" Oct 30 22:18:47.864169 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.864152 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.916546 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.916497 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c60cfe-openshift-default-dockercfg-6fplq\"" Oct 30 22:18:47.969132 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.969087 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb"] Oct 30 22:18:47.994784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.994742 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.994840 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.994913 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.994969 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995018 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.995003 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995275 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.995024 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkctm\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995275 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.995135 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995275 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.995176 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:47.995275 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:47.995260 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.096572 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096535 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.096758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096608 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.096758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096667 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.096758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096711 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.096758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096745 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097015 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096772 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097015 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096945 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097015 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.096997 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkctm\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097185 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.097084 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097185 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.097125 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097185 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.097141 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097341 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.097236 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097441 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.097407 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.097662 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.097643 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.099214 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.099191 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.099504 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.099484 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.159902 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.159841 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.159902 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.159902 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkctm\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm\") pod \"e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.176952 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.176919 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:48.602413 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.602389 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb"] Oct 30 22:18:48.604831 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:18:48.604803 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69da9c71_cbd8_470f_8a1a_b65382124f58.slice/crio-8ae4c6f2fea26f60a94e407b88338958f6d56ad60ab426e387f94a82cdf90ef8 WatchSource:0}: Error finding container 8ae4c6f2fea26f60a94e407b88338958f6d56ad60ab426e387f94a82cdf90ef8: Status 404 returned error can't find the container with id 8ae4c6f2fea26f60a94e407b88338958f6d56ad60ab426e387f94a82cdf90ef8 Oct 30 22:18:48.606911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.606852 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:18:48.606974 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.606942 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:18:48.607012 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:48.606975 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:18:49.599308 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:49.599271 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" event={"ID":"69da9c71-cbd8-470f-8a1a-b65382124f58","Type":"ContainerStarted","Data":"ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8"} Oct 30 22:18:49.599735 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:49.599328 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" event={"ID":"69da9c71-cbd8-470f-8a1a-b65382124f58","Type":"ContainerStarted","Data":"8ae4c6f2fea26f60a94e407b88338958f6d56ad60ab426e387f94a82cdf90ef8"} Oct 30 22:18:49.716260 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:49.716206 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podStartSLOduration=2.7161906780000002 podStartE2EDuration="2.716190678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:49.714862595 +0000 UTC m=+3681.616726703" watchObservedRunningTime="2026-10-30 22:18:49.716190678 +0000 UTC m=+3681.618054817" Oct 30 22:18:50.177461 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:50.177419 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:50.182575 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:50.182545 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:50.604992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:50.604911 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:18:50.605927 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:18:50.605906 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:19:12.345471 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:19:12.345422 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:19:14.493399 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.493365 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z"] Oct 30 22:19:14.497041 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.497025 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:14.554746 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.554714 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c60cfe\"" Oct 30 22:19:14.618910 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.618216 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7db25d04-dockercfg-tpmjl\"" Oct 30 22:19:14.620177 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.620150 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z"] Oct 30 22:19:14.675322 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.675288 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49r4p\" (UniqueName: \"kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p\") pod \"payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:14.675525 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.675430 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:14.776645 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.776558 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49r4p\" (UniqueName: \"kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p\") pod \"payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:14.776645 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.776644 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:14.777506 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.777476 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:14.837557 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:14.837521 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49r4p\" (UniqueName: \"kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p\") pod \"payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:15.112148 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:15.112050 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:15.487505 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:15.487461 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z"] Oct 30 22:19:15.487686 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:19:15.487656 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8af6b1_2f13_4e40_95e1_c7fd87094fee.slice/crio-3a374e1bdb2c566db2c740f401906d86753e7bb072d763fd52be55f9e21b091d WatchSource:0}: Error finding container 3a374e1bdb2c566db2c740f401906d86753e7bb072d763fd52be55f9e21b091d: Status 404 returned error can't find the container with id 3a374e1bdb2c566db2c740f401906d86753e7bb072d763fd52be55f9e21b091d Oct 30 22:19:15.732038 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:15.732003 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" event={"ID":"5e8af6b1-2f13-4e40-95e1-c7fd87094fee","Type":"ContainerStarted","Data":"b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3"} Oct 30 22:19:15.732486 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:15.732066 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:15.732486 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:15.732087 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" event={"ID":"5e8af6b1-2f13-4e40-95e1-c7fd87094fee","Type":"ContainerStarted","Data":"3a374e1bdb2c566db2c740f401906d86753e7bb072d763fd52be55f9e21b091d"} Oct 30 22:19:15.848300 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:15.848188 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" podStartSLOduration=1.848174582 podStartE2EDuration="1.848174582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:15.846947956 +0000 UTC m=+3707.748812021" watchObservedRunningTime="2026-10-30 22:19:15.848174582 +0000 UTC m=+3707.750038646" Oct 30 22:19:16.873166 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.873126 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv"] Oct 30 22:19:16.881613 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.878872 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:16.932335 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.932301 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c60cfe-serving-cert\"" Oct 30 22:19:16.986723 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.986690 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv"] Oct 30 22:19:16.997638 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.997605 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:16.997827 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.997655 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:16.997827 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.997798 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbz4\" (UniqueName: \"kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:16.997958 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:16.997845 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.099626 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.099581 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbz4\" (UniqueName: \"kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.099828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.099647 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.099828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.099752 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.099828 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:19:17.099778 5914 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-serving-cert: secret "maas-api-e2e-worker-single-main-c60cfe-serving-cert" not found Oct 30 22:19:17.099828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.099807 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.100033 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:19:17.099868 5914 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls podName:603d2c8f-3d45-4c92-bebb-58c28c537e30 nodeName:}" failed. No retries permitted until 2026-10-30 22:19:17.59984611 +0000 UTC m=+3709.501710166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls") pod "maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" (UID: "603d2c8f-3d45-4c92-bebb-58c28c537e30") : secret "maas-api-e2e-worker-single-main-c60cfe-serving-cert" not found Oct 30 22:19:17.100437 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.100414 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.102379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.102356 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.159973 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.159897 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbz4\" (UniqueName: \"kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.604652 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.604564 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.607231 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.607209 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:17.794331 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:17.794293 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:18.178207 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:18.178180 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv"] Oct 30 22:19:18.180058 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:19:18.180026 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603d2c8f_3d45_4c92_bebb_58c28c537e30.slice/crio-4152b2e634014653502c73bbd517d83e0e763672631e79fcdfb116f2d979d553 WatchSource:0}: Error finding container 4152b2e634014653502c73bbd517d83e0e763672631e79fcdfb116f2d979d553: Status 404 returned error can't find the container with id 4152b2e634014653502c73bbd517d83e0e763672631e79fcdfb116f2d979d553 Oct 30 22:19:18.181631 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:18.181610 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:19:18.747047 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:18.747011 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" event={"ID":"603d2c8f-3d45-4c92-bebb-58c28c537e30","Type":"ContainerStarted","Data":"4152b2e634014653502c73bbd517d83e0e763672631e79fcdfb116f2d979d553"} Oct 30 22:19:19.754236 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:19.754194 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" event={"ID":"603d2c8f-3d45-4c92-bebb-58c28c537e30","Type":"ContainerStarted","Data":"23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6"} Oct 30 22:19:19.754920 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:19.754257 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:19.871094 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:19.871043 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" podStartSLOduration=2.9367891139999998 podStartE2EDuration="3.871027371s" totalImagesPullingTime="934.238257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:19.869139043 +0000 UTC m=+3711.771003146" watchObservedRunningTime="2026-10-30 22:19:19.871027371 +0000 UTC m=+3711.772891435" Oct 30 22:19:25.127079 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:25.127043 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:19:25.766162 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:25.766133 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:19:26.739916 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:26.739869 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:19:34.308039 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:19:34.308006 5914 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69da9c71_cbd8_470f_8a1a_b65382124f58.slice/crio-conmon-ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:19:34.842780 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:34.842744 5914 generic.go:356] "Generic (PLEG): container finished" podID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerID="ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8" exitCode=0 Oct 30 22:19:34.842989 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:34.842816 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" event={"ID":"69da9c71-cbd8-470f-8a1a-b65382124f58","Type":"ContainerDied","Data":"ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8"} Oct 30 22:19:34.843234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:34.843220 5914 scope.go:122] "RemoveContainer" containerID="ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8" Oct 30 22:19:34.844477 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:34.844446 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:34.844577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:34.844518 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:34.844577 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:34.844550 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:19:35.605817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:35.605780 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:19:35.850743 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:35.850708 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" event={"ID":"69da9c71-cbd8-470f-8a1a-b65382124f58","Type":"ContainerStarted","Data":"805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc"} Oct 30 22:19:36.188907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:36.181097 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:19:36.188907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:36.181222 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:19:36.188907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:36.181266 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:19:37.191909 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:37.190357 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:19:37.191909 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:37.190450 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:19:38.198341 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:38.198296 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:19:38.198797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:38.198383 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:19:39.180356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:39.180314 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:19:39.180661 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:39.180625 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:19:40.192906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:40.188320 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:19:40.192906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:40.188384 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 22:19:41.181594 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:41.181562 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:19:41.181918 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:41.181848 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:19:41.182675 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:41.182658 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:19:42.345498 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:19:42.345455 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:19:48.859916 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.859869 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7"] Oct 30 22:19:48.863747 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.863729 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:48.916065 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.916036 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c60cfe\"/\"simulator-single-main-c60cfe-kserve-self-signed-certs\"" Oct 30 22:19:48.965734 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.965695 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c60cfe\"/\"default-dockercfg-fpkft\"" Oct 30 22:19:48.965982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.965831 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c60cfe\"/\"kube-root-ca.crt\"" Oct 30 22:19:48.965982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.965930 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c60cfe\"/\"openshift-service-ca.crt\"" Oct 30 22:19:48.967687 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.967662 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7"] Oct 30 22:19:48.977230 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.977210 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:48.977338 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.977253 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xxg\" (UniqueName: \"kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:48.977405 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.977377 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:48.977476 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.977420 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:48.977476 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.977468 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:48.977583 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:48.977562 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.078672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.078635 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.078672 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.078677 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.078930 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.078719 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.078930 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.078777 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.078930 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.078808 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xxg\" (UniqueName: \"kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.078930 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.078853 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.079146 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.079121 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.079210 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.079155 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.079251 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.079231 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.081313 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.081289 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.081461 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.081440 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.137165 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.137134 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xxg\" (UniqueName: \"kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg\") pod \"simulator-single-main-c60cfe-kserve-9d967484c-7pvj7\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.191745 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.191704 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:49.567696 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.564146 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7"] Oct 30 22:19:49.567696 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:19:49.567488 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56ab9f43_4059_4a05_9f4e_b7358f925fbb.slice/crio-57d8e6d07969b8e7a33ec4d06a5751f6cf6482b4087df792cdf046d363a526a2 WatchSource:0}: Error finding container 57d8e6d07969b8e7a33ec4d06a5751f6cf6482b4087df792cdf046d363a526a2: Status 404 returned error can't find the container with id 57d8e6d07969b8e7a33ec4d06a5751f6cf6482b4087df792cdf046d363a526a2 Oct 30 22:19:49.929472 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.929422 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" event={"ID":"56ab9f43-4059-4a05-9f4e-b7358f925fbb","Type":"ContainerStarted","Data":"8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238"} Oct 30 22:19:49.929472 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:49.929481 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" event={"ID":"56ab9f43-4059-4a05-9f4e-b7358f925fbb","Type":"ContainerStarted","Data":"57d8e6d07969b8e7a33ec4d06a5751f6cf6482b4087df792cdf046d363a526a2"} Oct 30 22:19:50.046101 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:50.046044 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" podStartSLOduration=2.046028149 podStartE2EDuration="2.046028149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:50.045012128 +0000 UTC m=+3741.946876194" watchObservedRunningTime="2026-10-30 22:19:50.046028149 +0000 UTC m=+3741.947892212" Oct 30 22:19:50.192712 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:50.192624 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:53.616164 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:53.616125 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:19:59.191981 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:59.191948 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:59.204354 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:59.204331 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:19:59.992244 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:19:59.992215 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:20:12.345269 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:20:12.345230 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:20:32.288817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:20:32.288778 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:20:42.345639 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:20:42.345554 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:20:59.410291 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:20:59.410252 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:21:02.516063 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:02.516029 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7"] Oct 30 22:21:09.649753 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:09.649718 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z"] Oct 30 22:21:12.345828 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:12.345784 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:21:14.003687 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:14.003646 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv"] Oct 30 22:21:14.004200 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:14.003923 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" podUID="603d2c8f-3d45-4c92-bebb-58c28c537e30" containerName="maas-api" containerID="cri-o://23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6" gracePeriod=30 Oct 30 22:21:14.650207 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:14.650165 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerName="payload-pre-processing" containerID="cri-o://b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3" gracePeriod=25 Oct 30 22:21:15.254610 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.254587 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:21:15.364851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.364750 5914 generic.go:356] "Generic (PLEG): container finished" podID="603d2c8f-3d45-4c92-bebb-58c28c537e30" containerID="23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6" exitCode=0 Oct 30 22:21:15.364851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.364793 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" Oct 30 22:21:15.364851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.364836 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" event={"ID":"603d2c8f-3d45-4c92-bebb-58c28c537e30","Type":"ContainerDied","Data":"23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6"} Oct 30 22:21:15.365151 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.364924 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv" event={"ID":"603d2c8f-3d45-4c92-bebb-58c28c537e30","Type":"ContainerDied","Data":"4152b2e634014653502c73bbd517d83e0e763672631e79fcdfb116f2d979d553"} Oct 30 22:21:15.365151 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.364950 5914 scope.go:122] "RemoveContainer" containerID="23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6" Oct 30 22:21:15.377044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.377022 5914 scope.go:122] "RemoveContainer" containerID="23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6" Oct 30 22:21:15.377476 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:15.377451 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6\": container with ID starting with 23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6 not found: ID does not exist" containerID="23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6" Oct 30 22:21:15.377606 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.377485 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6"} err="failed to get container status \"23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6\": rpc error: code = NotFound desc = could not find container \"23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6\": container with ID starting with 23a268eea7a4ee73ef8f39eec97b5793528cff1513b4edc2975d0d148db388b6 not found: ID does not exist" Oct 30 22:21:15.408963 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.408913 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs\") pod \"603d2c8f-3d45-4c92-bebb-58c28c537e30\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " Oct 30 22:21:15.409218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.409012 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4\" (UniqueName: \"kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4\") pod \"603d2c8f-3d45-4c92-bebb-58c28c537e30\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " Oct 30 22:21:15.409218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.409051 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\") pod \"603d2c8f-3d45-4c92-bebb-58c28c537e30\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " Oct 30 22:21:15.409218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.409122 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca\") pod \"603d2c8f-3d45-4c92-bebb-58c28c537e30\" (UID: \"603d2c8f-3d45-4c92-bebb-58c28c537e30\") " Oct 30 22:21:15.409860 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.409830 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca" pod "603d2c8f-3d45-4c92-bebb-58c28c537e30" (UID: "603d2c8f-3d45-4c92-bebb-58c28c537e30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:15.411479 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.411457 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs" pod "603d2c8f-3d45-4c92-bebb-58c28c537e30" (UID: "603d2c8f-3d45-4c92-bebb-58c28c537e30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:15.411479 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.411459 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4" pod "603d2c8f-3d45-4c92-bebb-58c28c537e30" (UID: "603d2c8f-3d45-4c92-bebb-58c28c537e30"). InnerVolumeSpecName "kube-api-access-mcbz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:15.411654 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.411521 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls" pod "603d2c8f-3d45-4c92-bebb-58c28c537e30" (UID: "603d2c8f-3d45-4c92-bebb-58c28c537e30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:15.510228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.510196 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.510228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.510225 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603d2c8f-3d45-4c92-bebb-58c28c537e30-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.510228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.510234 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603d2c8f-3d45-4c92-bebb-58c28c537e30-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.510486 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.510242 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbz4\" (UniqueName: \"kubernetes.io/projected/603d2c8f-3d45-4c92-bebb-58c28c537e30-kube-api-access-mcbz4\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.785191 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.785147 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv"] Oct 30 22:21:15.834897 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:15.834853 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c60cfe-684467fbc5-w58vv"] Oct 30 22:21:16.223002 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:16.222965 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603d2c8f-3d45-4c92-bebb-58c28c537e30" path="/var/lib/kubelet/pods/603d2c8f-3d45-4c92-bebb-58c28c537e30/volumes" Oct 30 22:21:16.739224 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:16.739186 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:21:17.536909 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.536839 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" podUID="56ab9f43-4059-4a05-9f4e-b7358f925fbb" containerName="main" containerID="cri-o://8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238" gracePeriod=45 Oct 30 22:21:17.795873 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.795806 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:21:17.939156 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939116 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg\" (UniqueName: \"kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg\") pod \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " Oct 30 22:21:17.939359 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939180 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs\") pod \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " Oct 30 22:21:17.939359 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939322 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir\") pod \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " Oct 30 22:21:17.939490 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939425 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm\") pod \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " Oct 30 22:21:17.939490 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939463 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache\") pod \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " Oct 30 22:21:17.939596 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939522 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home\") pod \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\" (UID: \"56ab9f43-4059-4a05-9f4e-b7358f925fbb\") " Oct 30 22:21:17.939657 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939610 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir" pod "56ab9f43-4059-4a05-9f4e-b7358f925fbb" (UID: "56ab9f43-4059-4a05-9f4e-b7358f925fbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:17.939991 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.939825 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache" pod "56ab9f43-4059-4a05-9f4e-b7358f925fbb" (UID: "56ab9f43-4059-4a05-9f4e-b7358f925fbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:17.940117 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.940003 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:17.940117 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.940028 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:17.940532 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.940491 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home" pod "56ab9f43-4059-4a05-9f4e-b7358f925fbb" (UID: "56ab9f43-4059-4a05-9f4e-b7358f925fbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:17.941982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.941957 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm" pod "56ab9f43-4059-4a05-9f4e-b7358f925fbb" (UID: "56ab9f43-4059-4a05-9f4e-b7358f925fbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:17.942245 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.942214 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg" pod "56ab9f43-4059-4a05-9f4e-b7358f925fbb" (UID: "56ab9f43-4059-4a05-9f4e-b7358f925fbb"). InnerVolumeSpecName "kube-api-access-s2xxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:17.942373 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:17.942258 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs" pod "56ab9f43-4059-4a05-9f4e-b7358f925fbb" (UID: "56ab9f43-4059-4a05-9f4e-b7358f925fbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:18.041398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.041359 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:18.041398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.041393 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xxg\" (UniqueName: \"kubernetes.io/projected/56ab9f43-4059-4a05-9f4e-b7358f925fbb-kube-api-access-s2xxg\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:18.041398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.041404 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56ab9f43-4059-4a05-9f4e-b7358f925fbb-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:18.041650 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.041412 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56ab9f43-4059-4a05-9f4e-b7358f925fbb-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:18.383104 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.383065 5914 generic.go:356] "Generic (PLEG): container finished" podID="56ab9f43-4059-4a05-9f4e-b7358f925fbb" containerID="8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238" exitCode=0 Oct 30 22:21:18.383287 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.383116 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" Oct 30 22:21:18.383287 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.383147 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" event={"ID":"56ab9f43-4059-4a05-9f4e-b7358f925fbb","Type":"ContainerDied","Data":"8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238"} Oct 30 22:21:18.383287 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.383195 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7" event={"ID":"56ab9f43-4059-4a05-9f4e-b7358f925fbb","Type":"ContainerDied","Data":"57d8e6d07969b8e7a33ec4d06a5751f6cf6482b4087df792cdf046d363a526a2"} Oct 30 22:21:18.383287 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.383213 5914 scope.go:122] "RemoveContainer" containerID="8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238" Oct 30 22:21:18.392520 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.392500 5914 scope.go:122] "RemoveContainer" containerID="8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238" Oct 30 22:21:18.392850 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:18.392823 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238\": container with ID starting with 8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238 not found: ID does not exist" containerID="8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238" Oct 30 22:21:18.392933 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.392854 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238"} err="failed to get container status \"8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238\": rpc error: code = NotFound desc = could not find container \"8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238\": container with ID starting with 8d6336d03dd675be641dcba60b017b43fa147567e3608b718256d27c8f681238 not found: ID does not exist" Oct 30 22:21:18.548410 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.548306 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7"] Oct 30 22:21:18.597355 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.597325 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c60cfe/simulator-single-main-c60cfe-kserve-9d967484c-7pvj7"] Oct 30 22:21:18.775278 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:18.775244 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:21:20.217580 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:20.217538 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ab9f43-4059-4a05-9f4e-b7358f925fbb" path="/var/lib/kubelet/pods/56ab9f43-4059-4a05-9f4e-b7358f925fbb/volumes" Oct 30 22:21:23.124711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:23.124667 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb"] Oct 30 22:21:23.125280 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:23.124984 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" containerID="cri-o://805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc" gracePeriod=30 Oct 30 22:21:26.184657 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:26.184614 5914 patch_prober.go:29] interesting pod/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:26.185148 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:26.184692 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:26.738522 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:26.738479 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:21:28.393006 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.392941 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:21:28.439822 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.439785 5914 generic.go:356] "Generic (PLEG): container finished" podID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerID="805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc" exitCode=0 Oct 30 22:21:28.440042 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.439836 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" Oct 30 22:21:28.440042 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.439862 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" event={"ID":"69da9c71-cbd8-470f-8a1a-b65382124f58","Type":"ContainerDied","Data":"805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc"} Oct 30 22:21:28.440042 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.439945 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb" event={"ID":"69da9c71-cbd8-470f-8a1a-b65382124f58","Type":"ContainerDied","Data":"8ae4c6f2fea26f60a94e407b88338958f6d56ad60ab426e387f94a82cdf90ef8"} Oct 30 22:21:28.440042 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.439973 5914 scope.go:122] "RemoveContainer" containerID="805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc" Oct 30 22:21:28.449256 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.449235 5914 scope.go:122] "RemoveContainer" containerID="ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8" Oct 30 22:21:28.453804 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.453773 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.453951 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.453832 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.453951 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.453908 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.453951 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.453946 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.454130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454037 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.454130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454086 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.454130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454096 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:28.454130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454124 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.454339 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454181 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.454339 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454219 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data\") pod \"69da9c71-cbd8-470f-8a1a-b65382124f58\" (UID: \"69da9c71-cbd8-470f-8a1a-b65382124f58\") " Oct 30 22:21:28.455044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454579 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.455044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454749 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:28.455044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454793 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:28.455044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454820 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:28.455044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.454830 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:28.456594 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.456554 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:28.456705 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.456648 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:28.458361 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.458333 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "kube-api-access-rkctm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:28.458742 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.458716 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token" pod "69da9c71-cbd8-470f-8a1a-b65382124f58" (UID: "69da9c71-cbd8-470f-8a1a-b65382124f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:28.460517 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.460497 5914 scope.go:122] "RemoveContainer" containerID="805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc" Oct 30 22:21:28.460872 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:28.460846 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc\": container with ID starting with 805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc not found: ID does not exist" containerID="805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc" Oct 30 22:21:28.461008 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.460953 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc"} err="failed to get container status \"805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc\": rpc error: code = NotFound desc = could not find container \"805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc\": container with ID starting with 805dc1f6a4dc1e0b63e7c1c784637ab8ca940c3e52b03e94aa9b78ea729288bc not found: ID does not exist" Oct 30 22:21:28.461008 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.460980 5914 scope.go:122] "RemoveContainer" containerID="ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8" Oct 30 22:21:28.461312 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:28.461293 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8\": container with ID starting with ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8 not found: ID does not exist" containerID="ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8" Oct 30 22:21:28.461360 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.461319 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8"} err="failed to get container status \"ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8\": rpc error: code = NotFound desc = could not find container \"ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8\": container with ID starting with ea2e8826afbe6e55611fd81a16c4440ec3648c1946894c48de02f8b7802453e8 not found: ID does not exist" Oct 30 22:21:28.555790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555745 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.555790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555782 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.555790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555795 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.556097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555807 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.556097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555822 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.556097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555834 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69da9c71-cbd8-470f-8a1a-b65382124f58-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.556097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555846 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkctm\" (UniqueName: \"kubernetes.io/projected/69da9c71-cbd8-470f-8a1a-b65382124f58-kube-api-access-rkctm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.556097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.555859 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69da9c71-cbd8-470f-8a1a-b65382124f58-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:28.859927 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.859859 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb"] Oct 30 22:21:28.865443 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:28.865407 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c60cfe-openshift-default-6974d657c-7hlfb"] Oct 30 22:21:29.534584 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.534548 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r"] Oct 30 22:21:29.535234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535213 5914 cpu_manager.go:414] "containerMap: removing container" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535237 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535273 5914 cpu_manager.go:414] "containerMap: removing container" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535281 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535295 5914 cpu_manager.go:414] "containerMap: removing container" podUID="56ab9f43-4059-4a05-9f4e-b7358f925fbb" containerName="main" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535303 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ab9f43-4059-4a05-9f4e-b7358f925fbb" containerName="main" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535344 5914 cpu_manager.go:414] "containerMap: removing container" podUID="603d2c8f-3d45-4c92-bebb-58c28c537e30" containerName="maas-api" Oct 30 22:21:29.535395 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535351 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603d2c8f-3d45-4c92-bebb-58c28c537e30" containerName="maas-api" Oct 30 22:21:29.535858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535458 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" Oct 30 22:21:29.535858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535477 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" containerName="istio-proxy" Oct 30 22:21:29.535858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535491 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ab9f43-4059-4a05-9f4e-b7358f925fbb" containerName="main" Oct 30 22:21:29.535858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.535507 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="603d2c8f-3d45-4c92-bebb-58c28c537e30" containerName="maas-api" Oct 30 22:21:29.540199 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.540177 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.565214 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.565183 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.565382 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.565328 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.565382 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.565368 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfbj\" (UniqueName: \"kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.565493 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.565465 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.565591 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.565574 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.565661 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.565648 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.594477 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.594445 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2841169b-kserve-self-signed-certs\"" Oct 30 22:21:29.645647 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.645613 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r"] Oct 30 22:21:29.667019 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.666970 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667216 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667025 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfbj\" (UniqueName: \"kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667216 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667055 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667216 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667106 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667216 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667144 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667216 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667171 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667513 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667486 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667553 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667513 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.667586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.667550 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.669543 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.669517 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.669681 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.669666 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.724234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.724201 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfbj\" (UniqueName: \"kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj\") pod \"e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:29.864346 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:29.864246 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:30.216588 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:30.216554 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69da9c71-cbd8-470f-8a1a-b65382124f58" path="/var/lib/kubelet/pods/69da9c71-cbd8-470f-8a1a-b65382124f58/volumes" Oct 30 22:21:30.243288 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:30.243264 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r"] Oct 30 22:21:30.245503 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:21:30.245473 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b0730e3_3c80_4d85_85f0_b887aad428e0.slice/crio-2f4c41fcb74a178f16babeda10fb139eb7cc30cf7cc26ee96eb2d58936c7259b WatchSource:0}: Error finding container 2f4c41fcb74a178f16babeda10fb139eb7cc30cf7cc26ee96eb2d58936c7259b: Status 404 returned error can't find the container with id 2f4c41fcb74a178f16babeda10fb139eb7cc30cf7cc26ee96eb2d58936c7259b Oct 30 22:21:30.453040 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:30.453004 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" event={"ID":"9b0730e3-3c80-4d85-85f0-b887aad428e0","Type":"ContainerStarted","Data":"be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647"} Oct 30 22:21:30.453234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:30.453047 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" event={"ID":"9b0730e3-3c80-4d85-85f0-b887aad428e0","Type":"ContainerStarted","Data":"2f4c41fcb74a178f16babeda10fb139eb7cc30cf7cc26ee96eb2d58936c7259b"} Oct 30 22:21:30.567218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:30.567162 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" podStartSLOduration=1.567147149 podStartE2EDuration="1.567147149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:30.566346705 +0000 UTC m=+3842.468210781" watchObservedRunningTime="2026-10-30 22:21:30.567147149 +0000 UTC m=+3842.469011213" Oct 30 22:21:30.864554 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:30.864464 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:34.203544 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:34.203503 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:21:34.955715 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:34.955685 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:21:35.024357 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.024317 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p\" (UniqueName: \"kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p\") pod \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " Oct 30 22:21:35.024589 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.024512 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume\") pod \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\" (UID: \"5e8af6b1-2f13-4e40-95e1-c7fd87094fee\") " Oct 30 22:21:35.025508 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.025460 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume" pod "5e8af6b1-2f13-4e40-95e1-c7fd87094fee" (UID: "5e8af6b1-2f13-4e40-95e1-c7fd87094fee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:35.027703 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.027670 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p" pod "5e8af6b1-2f13-4e40-95e1-c7fd87094fee" (UID: "5e8af6b1-2f13-4e40-95e1-c7fd87094fee"). InnerVolumeSpecName "kube-api-access-49r4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:35.126169 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.126126 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:35.126169 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.126169 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49r4p\" (UniqueName: \"kubernetes.io/projected/5e8af6b1-2f13-4e40-95e1-c7fd87094fee-kube-api-access-49r4p\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:35.482069 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.482028 5914 generic.go:356] "Generic (PLEG): container finished" podID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerID="b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3" exitCode=0 Oct 30 22:21:35.482593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.482102 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" Oct 30 22:21:35.482593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.482123 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" event={"ID":"5e8af6b1-2f13-4e40-95e1-c7fd87094fee","Type":"ContainerDied","Data":"b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3"} Oct 30 22:21:35.482593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.482198 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z" event={"ID":"5e8af6b1-2f13-4e40-95e1-c7fd87094fee","Type":"ContainerDied","Data":"3a374e1bdb2c566db2c740f401906d86753e7bb072d763fd52be55f9e21b091d"} Oct 30 22:21:35.482593 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.482232 5914 scope.go:122] "RemoveContainer" containerID="b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3" Oct 30 22:21:35.504509 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.504481 5914 scope.go:122] "RemoveContainer" containerID="b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3" Oct 30 22:21:35.509656 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:35.509622 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3\": container with ID starting with b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3 not found: ID does not exist" containerID="b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3" Oct 30 22:21:35.509799 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.509668 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3"} err="failed to get container status \"b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3\": rpc error: code = NotFound desc = could not find container \"b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3\": container with ID starting with b82a46b3a9a049a292995a746d19b4375d1f165a8d67a8dfb2b79d2f9ee605c3 not found: ID does not exist" Oct 30 22:21:35.649244 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.649190 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z"] Oct 30 22:21:35.655045 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:35.655013 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c60cfe-d867dd4s2z"] Oct 30 22:21:36.258911 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:36.250916 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" path="/var/lib/kubelet/pods/5e8af6b1-2f13-4e40-95e1-c7fd87094fee/volumes" Oct 30 22:21:39.864629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:39.864592 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:39.877555 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:39.877526 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:40.553492 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:21:40.553460 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:21:42.344960 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:21:42.344922 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:22:11.770629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.770586 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x"] Oct 30 22:22:11.771295 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.771273 5914 cpu_manager.go:414] "containerMap: removing container" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerName="payload-pre-processing" Oct 30 22:22:11.771393 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.771298 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerName="payload-pre-processing" Oct 30 22:22:11.771470 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.771457 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8af6b1-2f13-4e40-95e1-c7fd87094fee" containerName="payload-pre-processing" Oct 30 22:22:11.774643 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.774622 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.878832 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.878799 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2841169b-kserve-self-signed-certs\"" Oct 30 22:22:11.880297 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.880277 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x"] Oct 30 22:22:11.894536 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.894502 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.894713 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.894566 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmswb\" (UniqueName: \"kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.894769 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.894733 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.894820 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.894767 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.894925 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.894907 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.894970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.894935 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.995581 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.995536 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmswb\" (UniqueName: \"kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.995806 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.995601 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.995806 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.995640 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.995806 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.995680 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.995806 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.995697 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.995806 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.995780 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.996159 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.996096 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.996222 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.996163 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.996222 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.996181 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.998250 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.998224 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:11.998347 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:11.998322 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:12.053687 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:12.053618 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmswb\" (UniqueName: \"kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb\") pod \"e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:12.086841 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:12.086803 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:12.345414 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:22:12.345330 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:22:12.463201 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:12.463178 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x"] Oct 30 22:22:12.465728 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:22:12.465693 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4ae9989_7bcc_4fe1_b9b4_d50cadf04451.slice/crio-efbfce49ad6d3e57d5e8a3b3a9c1cbccba74b84a64597bd3b9efad559a6571ab WatchSource:0}: Error finding container efbfce49ad6d3e57d5e8a3b3a9c1cbccba74b84a64597bd3b9efad559a6571ab: Status 404 returned error can't find the container with id efbfce49ad6d3e57d5e8a3b3a9c1cbccba74b84a64597bd3b9efad559a6571ab Oct 30 22:22:12.706754 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:12.706714 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" event={"ID":"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451","Type":"ContainerStarted","Data":"89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a"} Oct 30 22:22:12.706963 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:12.706765 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" event={"ID":"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451","Type":"ContainerStarted","Data":"efbfce49ad6d3e57d5e8a3b3a9c1cbccba74b84a64597bd3b9efad559a6571ab"} Oct 30 22:22:12.824011 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:12.823948 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" podStartSLOduration=1.823929432 podStartE2EDuration="1.823929432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:12.823820015 +0000 UTC m=+3884.725684080" watchObservedRunningTime="2026-10-30 22:22:12.823929432 +0000 UTC m=+3884.725793498" Oct 30 22:22:13.087782 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:13.087687 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:16.396615 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:16.396577 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:22:22.087381 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:22.087340 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:22.106044 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:22.106012 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:22.786442 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:22.786411 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:22:29.470403 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:29.470372 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:22:29.487066 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:29.487043 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:22:42.344980 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:22:42.344939 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:22:48.006330 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:22:48.006293 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:23:01.715387 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:01.715352 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:23:09.804575 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:09.804540 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r"] Oct 30 22:23:12.344247 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:23:12.344209 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:23:12.517751 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:12.517705 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:23:24.825658 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:24.825615 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" podUID="9b0730e3-3c80-4d85-85f0-b887aad428e0" containerName="main" containerID="cri-o://be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647" gracePeriod=45 Oct 30 22:23:25.085273 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.085206 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:23:25.124494 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124450 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache\") pod \"9b0730e3-3c80-4d85-85f0-b887aad428e0\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " Oct 30 22:23:25.124680 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124502 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir\") pod \"9b0730e3-3c80-4d85-85f0-b887aad428e0\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " Oct 30 22:23:25.124680 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124621 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs\") pod \"9b0730e3-3c80-4d85-85f0-b887aad428e0\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " Oct 30 22:23:25.124680 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124662 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home\") pod \"9b0730e3-3c80-4d85-85f0-b887aad428e0\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " Oct 30 22:23:25.124918 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124707 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm\") pod \"9b0730e3-3c80-4d85-85f0-b887aad428e0\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " Oct 30 22:23:25.124918 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124775 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache" pod "9b0730e3-3c80-4d85-85f0-b887aad428e0" (UID: "9b0730e3-3c80-4d85-85f0-b887aad428e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:25.124918 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124793 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj\" (UniqueName: \"kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj\") pod \"9b0730e3-3c80-4d85-85f0-b887aad428e0\" (UID: \"9b0730e3-3c80-4d85-85f0-b887aad428e0\") " Oct 30 22:23:25.125086 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124790 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir" pod "9b0730e3-3c80-4d85-85f0-b887aad428e0" (UID: "9b0730e3-3c80-4d85-85f0-b887aad428e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:25.125086 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.124979 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home" pod "9b0730e3-3c80-4d85-85f0-b887aad428e0" (UID: "9b0730e3-3c80-4d85-85f0-b887aad428e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:25.125226 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.125207 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.125285 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.125232 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.125285 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.125250 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.127054 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.127021 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs" pod "9b0730e3-3c80-4d85-85f0-b887aad428e0" (UID: "9b0730e3-3c80-4d85-85f0-b887aad428e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:25.127175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.127082 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm" pod "9b0730e3-3c80-4d85-85f0-b887aad428e0" (UID: "9b0730e3-3c80-4d85-85f0-b887aad428e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:25.127175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.127101 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj" pod "9b0730e3-3c80-4d85-85f0-b887aad428e0" (UID: "9b0730e3-3c80-4d85-85f0-b887aad428e0"). InnerVolumeSpecName "kube-api-access-lhfbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:25.186315 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.186275 5914 generic.go:356] "Generic (PLEG): container finished" podID="9b0730e3-3c80-4d85-85f0-b887aad428e0" containerID="be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647" exitCode=0 Oct 30 22:23:25.186499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.186325 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" Oct 30 22:23:25.186499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.186356 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" event={"ID":"9b0730e3-3c80-4d85-85f0-b887aad428e0","Type":"ContainerDied","Data":"be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647"} Oct 30 22:23:25.186499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.186396 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r" event={"ID":"9b0730e3-3c80-4d85-85f0-b887aad428e0","Type":"ContainerDied","Data":"2f4c41fcb74a178f16babeda10fb139eb7cc30cf7cc26ee96eb2d58936c7259b"} Oct 30 22:23:25.186499 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.186411 5914 scope.go:122] "RemoveContainer" containerID="be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647" Oct 30 22:23:25.196130 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.196108 5914 scope.go:122] "RemoveContainer" containerID="be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647" Oct 30 22:23:25.196412 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:23:25.196390 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647\": container with ID starting with be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647 not found: ID does not exist" containerID="be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647" Oct 30 22:23:25.196463 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.196421 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647"} err="failed to get container status \"be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647\": rpc error: code = NotFound desc = could not find container \"be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647\": container with ID starting with be4c035a491a6baf42185dc6165a57ff4adeec20307906f70712be9cbc6cc647 not found: ID does not exist" Oct 30 22:23:25.225991 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.225960 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfbj\" (UniqueName: \"kubernetes.io/projected/9b0730e3-3c80-4d85-85f0-b887aad428e0-kube-api-access-lhfbj\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.225991 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.225991 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0730e3-3c80-4d85-85f0-b887aad428e0-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.226203 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.226003 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0730e3-3c80-4d85-85f0-b887aad428e0-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:25.309632 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.309593 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r"] Oct 30 22:23:25.313224 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.313198 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2841169b-kserve-6459494c45-hjb5r"] Oct 30 22:23:25.884430 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:25.884390 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x"] Oct 30 22:23:26.215495 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:26.215413 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0730e3-3c80-4d85-85f0-b887aad428e0" path="/var/lib/kubelet/pods/9b0730e3-3c80-4d85-85f0-b887aad428e0/volumes" Oct 30 22:23:30.319823 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:30.319779 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:23:40.905723 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:40.905606 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" podUID="e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" containerName="main" containerID="cri-o://89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a" gracePeriod=45 Oct 30 22:23:41.175864 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.175836 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:23:41.324121 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.324085 5914 generic.go:356] "Generic (PLEG): container finished" podID="e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" containerID="89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a" exitCode=0 Oct 30 22:23:41.324306 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.324135 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" Oct 30 22:23:41.324306 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.324153 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" event={"ID":"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451","Type":"ContainerDied","Data":"89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a"} Oct 30 22:23:41.324306 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.324196 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x" event={"ID":"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451","Type":"ContainerDied","Data":"efbfce49ad6d3e57d5e8a3b3a9c1cbccba74b84a64597bd3b9efad559a6571ab"} Oct 30 22:23:41.324306 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.324218 5914 scope.go:122] "RemoveContainer" containerID="89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a" Oct 30 22:23:41.334176 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.334152 5914 scope.go:122] "RemoveContainer" containerID="89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a" Oct 30 22:23:41.334464 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:23:41.334443 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a\": container with ID starting with 89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a not found: ID does not exist" containerID="89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a" Oct 30 22:23:41.334534 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.334475 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a"} err="failed to get container status \"89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a\": rpc error: code = NotFound desc = could not find container \"89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a\": container with ID starting with 89de472ecd63a9878f926b8de54017d076ecc18600b714a3e764a10198f4368a not found: ID does not exist" Oct 30 22:23:41.350805 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.350776 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm\") pod \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " Oct 30 22:23:41.350959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.350818 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs\") pod \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " Oct 30 22:23:41.350959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.350853 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home\") pod \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " Oct 30 22:23:41.350959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.350934 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache\") pod \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " Oct 30 22:23:41.351139 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.350972 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir\") pod \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " Oct 30 22:23:41.351139 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.351026 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb\" (UniqueName: \"kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb\") pod \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\" (UID: \"e4ae9989-7bcc-4fe1-b9b4-d50cadf04451\") " Oct 30 22:23:41.351240 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.351153 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home" pod "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" (UID: "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.351295 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.351236 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache" pod "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" (UID: "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.351295 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.351246 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir" pod "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" (UID: "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.351484 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.351468 5914 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-model-cache\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.351553 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.351502 5914 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tmp-dir\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.353177 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.353153 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm" pod "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" (UID: "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.353291 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.353212 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs" pod "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" (UID: "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:23:41.353368 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.353342 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb" pod "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" (UID: "e4ae9989-7bcc-4fe1-b9b4-d50cadf04451"). InnerVolumeSpecName "kube-api-access-tmswb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:41.452956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.452843 5914 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-dshm\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.452956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.452873 5914 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-tls-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.452956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.452916 5914 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-home\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.452956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.452931 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmswb\" (UniqueName: \"kubernetes.io/projected/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451-kube-api-access-tmswb\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.744597 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.744509 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x"] Oct 30 22:23:41.748288 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:41.748261 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2841169b-kserve-69c8f94cd5-hll5x"] Oct 30 22:23:42.215217 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:42.215172 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" path="/var/lib/kubelet/pods/e4ae9989-7bcc-4fe1-b9b4-d50cadf04451/volumes" Oct 30 22:23:42.345593 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:23:42.345560 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:23:43.008819 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:43.008782 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:23:54.282962 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:23:54.282931 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:24:12.345348 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:24:12.345309 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:24:15.781095 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:15.781054 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:24:26.244609 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.244575 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5644488b47-cvjsn"] Oct 30 22:24:26.245086 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.245065 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" containerName="main" Oct 30 22:24:26.245086 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.245083 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" containerName="main" Oct 30 22:24:26.245244 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.245095 5914 cpu_manager.go:414] "containerMap: removing container" podUID="9b0730e3-3c80-4d85-85f0-b887aad428e0" containerName="main" Oct 30 22:24:26.245244 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.245101 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0730e3-3c80-4d85-85f0-b887aad428e0" containerName="main" Oct 30 22:24:26.245244 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.245200 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ae9989-7bcc-4fe1-b9b4-d50cadf04451" containerName="main" Oct 30 22:24:26.245244 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.245213 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0730e3-3c80-4d85-85f0-b887aad428e0" containerName="main" Oct 30 22:24:26.248897 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.248862 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.328324 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.328294 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c285e1-7da5-4065-a3dc-443854d0bf10-metrics-certs\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.328522 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.328330 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84gf\" (UniqueName: \"kubernetes.io/projected/d3c285e1-7da5-4065-a3dc-443854d0bf10-kube-api-access-z84gf\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.328522 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.328447 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c285e1-7da5-4065-a3dc-443854d0bf10-openshift-service-ca\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.328611 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.328556 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c285e1-7da5-4065-a3dc-443854d0bf10-maas-api-tls\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.354994 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.354962 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5644488b47-cvjsn"] Oct 30 22:24:26.430114 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.430078 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c285e1-7da5-4065-a3dc-443854d0bf10-openshift-service-ca\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.430312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.430139 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c285e1-7da5-4065-a3dc-443854d0bf10-maas-api-tls\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.430312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.430194 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c285e1-7da5-4065-a3dc-443854d0bf10-metrics-certs\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.430312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.430274 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84gf\" (UniqueName: \"kubernetes.io/projected/d3c285e1-7da5-4065-a3dc-443854d0bf10-kube-api-access-z84gf\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.430845 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.430820 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3c285e1-7da5-4065-a3dc-443854d0bf10-openshift-service-ca\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.432854 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.432828 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3c285e1-7da5-4065-a3dc-443854d0bf10-metrics-certs\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.432988 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.432865 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3c285e1-7da5-4065-a3dc-443854d0bf10-maas-api-tls\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.487396 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.487362 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84gf\" (UniqueName: \"kubernetes.io/projected/d3c285e1-7da5-4065-a3dc-443854d0bf10-kube-api-access-z84gf\") pod \"maas-api-5644488b47-cvjsn\" (UID: \"d3c285e1-7da5-4065-a3dc-443854d0bf10\") " pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.561463 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.561378 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:26.933006 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.932961 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5644488b47-cvjsn"] Oct 30 22:24:26.935855 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:24:26.935815 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c285e1_7da5_4065_a3dc_443854d0bf10.slice/crio-9f76c518d1765af9af0d7dd6de6d3c8f4b439b16c4d124a3045100a12e052f04 WatchSource:0}: Error finding container 9f76c518d1765af9af0d7dd6de6d3c8f4b439b16c4d124a3045100a12e052f04: Status 404 returned error can't find the container with id 9f76c518d1765af9af0d7dd6de6d3c8f4b439b16c4d124a3045100a12e052f04 Oct 30 22:24:26.937313 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:26.937294 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:24:27.663042 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:27.663001 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" event={"ID":"d3c285e1-7da5-4065-a3dc-443854d0bf10","Type":"ContainerStarted","Data":"9f76c518d1765af9af0d7dd6de6d3c8f4b439b16c4d124a3045100a12e052f04"} Oct 30 22:24:28.670144 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:28.670108 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" event={"ID":"d3c285e1-7da5-4065-a3dc-443854d0bf10","Type":"ContainerStarted","Data":"09d827c2bb2dd67c843b869e69a8885fba92c98af98ae96a003fa68d4098d2c4"} Oct 30 22:24:28.670571 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:28.670193 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:28.786415 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:28.786360 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" podStartSLOduration=1.937218883 podStartE2EDuration="2.786345646s" totalImagesPullingTime="849.126763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:28.785643426 +0000 UTC m=+4020.687507489" watchObservedRunningTime="2026-10-30 22:24:28.786345646 +0000 UTC m=+4020.688209745" Oct 30 22:24:33.300214 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.300176 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr"] Oct 30 22:24:33.304051 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.304026 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.358166 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.358119 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-daf0fd7e-openshift-default-dockercfg-4d8hj\"" Oct 30 22:24:33.406058 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406021 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406391 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406362 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406547 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406507 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406616 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406588 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406674 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406630 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406674 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406658 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvks\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406788 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406686 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.406910 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.406854 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.407155 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.407122 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.408047 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.408025 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr"] Oct 30 22:24:33.508736 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.508698 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.508970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.508742 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.508970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.508792 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.508970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.508818 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.508970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.508837 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.508970 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.508856 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvks\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509005 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509106 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509150 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509163 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509251 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509565 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509454 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509565 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509485 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.509732 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.509708 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.511480 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.511458 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.512129 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.512106 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.566648 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.566573 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.566648 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.566617 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvks\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks\") pod \"e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:33.617177 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:33.617138 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:34.042330 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.042302 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr"] Oct 30 22:24:34.044940 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:24:34.044912 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64b4dd5_a7a0_4d6a_ada9_4a97263b5b5e.slice/crio-fa5dba2302d2e6be96c29bf90eb64057c219d3ee781b017d52ce960e6095ad38 WatchSource:0}: Error finding container fa5dba2302d2e6be96c29bf90eb64057c219d3ee781b017d52ce960e6095ad38: Status 404 returned error can't find the container with id fa5dba2302d2e6be96c29bf90eb64057c219d3ee781b017d52ce960e6095ad38 Oct 30 22:24:34.047057 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.047026 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:34.047134 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.047095 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:34.047134 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.047126 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:34.680901 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.680855 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5644488b47-cvjsn" Oct 30 22:24:34.701289 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.701257 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" event={"ID":"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e","Type":"ContainerStarted","Data":"5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a"} Oct 30 22:24:34.701466 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.701317 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" event={"ID":"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e","Type":"ContainerStarted","Data":"fa5dba2302d2e6be96c29bf90eb64057c219d3ee781b017d52ce960e6095ad38"} Oct 30 22:24:34.913662 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:34.913602 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podStartSLOduration=1.913581158 podStartE2EDuration="1.913581158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:34.913161834 +0000 UTC m=+4026.815025931" watchObservedRunningTime="2026-10-30 22:24:34.913581158 +0000 UTC m=+4026.815445226" Oct 30 22:24:35.617869 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:35.617835 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:35.623092 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:35.623064 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:35.706185 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:35.706153 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:35.707314 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:35.707291 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:24:42.345517 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:24:42.345476 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:24:42.981232 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:42.981203 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-4zll5"] Oct 30 22:24:42.981437 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:42.981402 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" containerID="cri-o://0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b" gracePeriod=30 Oct 30 22:24:44.086732 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.086691 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": dial tcp 10.134.0.32:8443: connect: connection refused" Oct 30 22:24:44.241015 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.240989 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 22:24:44.322047 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.322015 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs\") pod \"d03b79e3-577c-4a1f-a90c-aa48807550b6\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " Oct 30 22:24:44.322232 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.322152 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l\" (UniqueName: \"kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l\") pod \"d03b79e3-577c-4a1f-a90c-aa48807550b6\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " Oct 30 22:24:44.322232 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.322198 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls\") pod \"d03b79e3-577c-4a1f-a90c-aa48807550b6\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " Oct 30 22:24:44.322343 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.322249 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca\") pod \"d03b79e3-577c-4a1f-a90c-aa48807550b6\" (UID: \"d03b79e3-577c-4a1f-a90c-aa48807550b6\") " Oct 30 22:24:44.322736 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.322703 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca" pod "d03b79e3-577c-4a1f-a90c-aa48807550b6" (UID: "d03b79e3-577c-4a1f-a90c-aa48807550b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:44.324353 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.324322 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs" pod "d03b79e3-577c-4a1f-a90c-aa48807550b6" (UID: "d03b79e3-577c-4a1f-a90c-aa48807550b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:44.324520 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.324498 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls" pod "d03b79e3-577c-4a1f-a90c-aa48807550b6" (UID: "d03b79e3-577c-4a1f-a90c-aa48807550b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:44.324568 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.324527 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l" pod "d03b79e3-577c-4a1f-a90c-aa48807550b6" (UID: "d03b79e3-577c-4a1f-a90c-aa48807550b6"). InnerVolumeSpecName "kube-api-access-sf65l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:44.424095 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.424059 5914 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-metrics-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.424095 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.424090 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf65l\" (UniqueName: \"kubernetes.io/projected/d03b79e3-577c-4a1f-a90c-aa48807550b6-kube-api-access-sf65l\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.424095 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.424100 5914 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b79e3-577c-4a1f-a90c-aa48807550b6-maas-api-tls\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.424353 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.424109 5914 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b79e3-577c-4a1f-a90c-aa48807550b6-openshift-service-ca\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:24:44.751605 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.751569 5914 generic.go:356] "Generic (PLEG): container finished" podID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerID="0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b" exitCode=0 Oct 30 22:24:44.751784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.751613 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" Oct 30 22:24:44.751784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.751654 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" event={"ID":"d03b79e3-577c-4a1f-a90c-aa48807550b6","Type":"ContainerDied","Data":"0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b"} Oct 30 22:24:44.751784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.751710 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-745b76d875-4zll5" event={"ID":"d03b79e3-577c-4a1f-a90c-aa48807550b6","Type":"ContainerDied","Data":"3f3378b7833a5e2c13e257a6f2d54f2c830f6f5dc24073918002c8b8094689af"} Oct 30 22:24:44.751784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.751733 5914 scope.go:122] "RemoveContainer" containerID="0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b" Oct 30 22:24:44.761996 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.761979 5914 scope.go:122] "RemoveContainer" containerID="4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c" Oct 30 22:24:44.773141 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.773122 5914 scope.go:122] "RemoveContainer" containerID="0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b" Oct 30 22:24:44.773448 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:24:44.773421 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b\": container with ID starting with 0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b not found: ID does not exist" containerID="0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b" Oct 30 22:24:44.773503 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.773458 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b"} err="failed to get container status \"0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b\": rpc error: code = NotFound desc = could not find container \"0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b\": container with ID starting with 0c28937e660e0eb683a4b8f801522aadab4ed7a6837f35b29d6ab06a3fa79a2b not found: ID does not exist" Oct 30 22:24:44.773503 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.773475 5914 scope.go:122] "RemoveContainer" containerID="4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c" Oct 30 22:24:44.773718 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:24:44.773696 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c\": container with ID starting with 4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c not found: ID does not exist" containerID="4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c" Oct 30 22:24:44.773761 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.773725 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c"} err="failed to get container status \"4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c\": rpc error: code = NotFound desc = could not find container \"4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c\": container with ID starting with 4ded836e04b07c1483add981614b957fb1ade4abb49dff5f501cd5e700bb7b2c not found: ID does not exist" Oct 30 22:24:44.914139 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.914104 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-4zll5"] Oct 30 22:24:44.963729 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:44.963694 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-745b76d875-4zll5"] Oct 30 22:24:46.216149 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:46.216110 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" path="/var/lib/kubelet/pods/d03b79e3-577c-4a1f-a90c-aa48807550b6/volumes" Oct 30 22:24:50.675977 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.675940 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545"] Oct 30 22:24:50.676587 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.676565 5914 cpu_manager.go:414] "containerMap: removing container" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" Oct 30 22:24:50.676657 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.676591 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" Oct 30 22:24:50.676756 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.676742 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" Oct 30 22:24:50.681177 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.681161 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.734639 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.734596 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-daf0fd7e\"" Oct 30 22:24:50.736945 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.736912 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545"] Oct 30 22:24:50.792375 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.792328 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.792718 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.792485 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgjw\" (UniqueName: \"kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw\") pod \"payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.829763 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.829730 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8"] Oct 30 22:24:50.830298 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.830282 5914 cpu_manager.go:414] "containerMap: removing container" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" Oct 30 22:24:50.830356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.830300 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" Oct 30 22:24:50.830397 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.830377 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03b79e3-577c-4a1f-a90c-aa48807550b6" containerName="maas-api" Oct 30 22:24:50.833552 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.833535 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:50.887676 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.887647 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-daf0fd7e-dockercfg-82ngr\"" Oct 30 22:24:50.889655 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.889630 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8"] Oct 30 22:24:50.893849 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.893820 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.894031 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.893872 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:50.894031 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.894014 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-769dx\" (UniqueName: \"kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx\") pod \"payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:50.894171 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.894154 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgjw\" (UniqueName: \"kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw\") pod \"payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.894640 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.894620 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.951153 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.951086 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgjw\" (UniqueName: \"kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw\") pod \"payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.994279 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.994241 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:50.994734 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.994713 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:50.994780 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.994762 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-769dx\" (UniqueName: \"kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx\") pod \"payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:50.995421 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:50.995403 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:51.054924 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.054558 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-769dx\" (UniqueName: \"kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx\") pod \"payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:51.147580 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.147539 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:51.365528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.365506 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545"] Oct 30 22:24:51.367834 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:24:51.367806 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bd7a95_7564_46fe_8c50_efd4f07fa364.slice/crio-c16cd87b95a97fd6b89790f4d3524ba00ffb97063d32d0bc2daf11c7093c40b8 WatchSource:0}: Error finding container c16cd87b95a97fd6b89790f4d3524ba00ffb97063d32d0bc2daf11c7093c40b8: Status 404 returned error can't find the container with id c16cd87b95a97fd6b89790f4d3524ba00ffb97063d32d0bc2daf11c7093c40b8 Oct 30 22:24:51.524576 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.524550 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8"] Oct 30 22:24:51.527145 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:24:51.527081 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a3428b_6271_472c_94ee_c4432dbb7c7d.slice/crio-dd436f3ed3b20bceec42d473a0a72f0d95d18ece147cdfb56442dac1159fd40a WatchSource:0}: Error finding container dd436f3ed3b20bceec42d473a0a72f0d95d18ece147cdfb56442dac1159fd40a: Status 404 returned error can't find the container with id dd436f3ed3b20bceec42d473a0a72f0d95d18ece147cdfb56442dac1159fd40a Oct 30 22:24:51.791961 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.791530 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" event={"ID":"e1a3428b-6271-472c-94ee-c4432dbb7c7d","Type":"ContainerStarted","Data":"2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55"} Oct 30 22:24:51.791961 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.791604 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" event={"ID":"e1a3428b-6271-472c-94ee-c4432dbb7c7d","Type":"ContainerStarted","Data":"dd436f3ed3b20bceec42d473a0a72f0d95d18ece147cdfb56442dac1159fd40a"} Oct 30 22:24:51.791961 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.791848 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:24:51.794562 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.794521 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" event={"ID":"19bd7a95-7564-46fe-8c50-efd4f07fa364","Type":"ContainerStarted","Data":"b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef"} Oct 30 22:24:51.794778 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.794587 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" event={"ID":"19bd7a95-7564-46fe-8c50-efd4f07fa364","Type":"ContainerStarted","Data":"c16cd87b95a97fd6b89790f4d3524ba00ffb97063d32d0bc2daf11c7093c40b8"} Oct 30 22:24:51.795253 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.795230 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:24:51.907123 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:51.907048 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" podStartSLOduration=1.907028622 podStartE2EDuration="1.907028622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:51.905354343 +0000 UTC m=+4043.807218408" watchObservedRunningTime="2026-10-30 22:24:51.907028622 +0000 UTC m=+4043.808892687" Oct 30 22:24:59.425086 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:59.425030 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" podStartSLOduration=9.425014012 podStartE2EDuration="9.425014012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:52.022738355 +0000 UTC m=+4043.924602420" watchObservedRunningTime="2026-10-30 22:24:59.425014012 +0000 UTC m=+4051.326878077" Oct 30 22:24:59.425978 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:24:59.425957 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:25:02.803537 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:02.803506 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:25:03.808282 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:03.808250 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:25:06.748168 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:06.748070 5914 patch_prober.go:29] interesting pod/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:25:06.748551 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:06.748160 5914 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:25:08.882223 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:08.882187 5914 generic.go:356] "Generic (PLEG): container finished" podID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerID="5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a" exitCode=0 Oct 30 22:25:08.882840 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:08.882240 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" event={"ID":"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e","Type":"ContainerDied","Data":"5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a"} Oct 30 22:25:08.882840 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:08.882773 5914 scope.go:122] "RemoveContainer" containerID="5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a" Oct 30 22:25:08.884099 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:08.884072 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:25:08.884179 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:08.884138 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:25:08.884179 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:08.884171 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:25:09.901629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:09.901587 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" event={"ID":"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e","Type":"ContainerStarted","Data":"15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f"} Oct 30 22:25:10.617956 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:10.617918 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:25:10.618251 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:10.618225 5914 patch_prober.go:29] interesting pod/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:25:10.618312 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:10.618283 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 22:25:11.618419 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:11.618384 5914 patch_prober.go:29] interesting pod/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:25:11.618835 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:11.618434 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 22:25:12.345730 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:25:12.345694 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:25:12.618286 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:12.618203 5914 patch_prober.go:29] interesting pod/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:25:12.618286 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:12.618255 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 22:25:13.617495 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:13.617462 5914 patch_prober.go:29] interesting pod/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:25:13.617786 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:13.617516 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 22:25:14.617848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:14.617813 5914 patch_prober.go:29] interesting pod/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 22:25:14.618308 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:14.617871 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 22:25:15.621829 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:15.621799 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:25:15.622416 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:15.622225 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:25:15.622999 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:15.622977 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:25:42.345579 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:25:42.345535 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:25:45.054992 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:45.054952 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8"] Oct 30 22:25:45.160702 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:45.160667 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545"] Oct 30 22:25:50.055628 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.055562 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" podUID="e1a3428b-6271-472c-94ee-c4432dbb7c7d" containerName="payload-processing" containerID="cri-o://2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55" gracePeriod=25 Oct 30 22:25:50.161869 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.161790 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" podUID="19bd7a95-7564-46fe-8c50-efd4f07fa364" containerName="payload-pre-processing" containerID="cri-o://b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef" gracePeriod=25 Oct 30 22:25:50.335855 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.335826 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:25:50.418771 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.418745 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:25:50.451703 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.451671 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume\") pod \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " Oct 30 22:25:50.451955 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.451837 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx\" (UniqueName: \"kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx\") pod \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\" (UID: \"e1a3428b-6271-472c-94ee-c4432dbb7c7d\") " Oct 30 22:25:50.452140 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.452114 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume" pod "e1a3428b-6271-472c-94ee-c4432dbb7c7d" (UID: "e1a3428b-6271-472c-94ee-c4432dbb7c7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:50.452323 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.452301 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a3428b-6271-472c-94ee-c4432dbb7c7d-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:50.454534 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.454507 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx" pod "e1a3428b-6271-472c-94ee-c4432dbb7c7d" (UID: "e1a3428b-6271-472c-94ee-c4432dbb7c7d"). InnerVolumeSpecName "kube-api-access-769dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:50.553352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.553319 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume\") pod \"19bd7a95-7564-46fe-8c50-efd4f07fa364\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " Oct 30 22:25:50.553540 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.553426 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw\" (UniqueName: \"kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw\") pod \"19bd7a95-7564-46fe-8c50-efd4f07fa364\" (UID: \"19bd7a95-7564-46fe-8c50-efd4f07fa364\") " Oct 30 22:25:50.553752 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.553737 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-769dx\" (UniqueName: \"kubernetes.io/projected/e1a3428b-6271-472c-94ee-c4432dbb7c7d-kube-api-access-769dx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:50.553804 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.553778 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume" pod "19bd7a95-7564-46fe-8c50-efd4f07fa364" (UID: "19bd7a95-7564-46fe-8c50-efd4f07fa364"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:50.555717 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.555694 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw" pod "19bd7a95-7564-46fe-8c50-efd4f07fa364" (UID: "19bd7a95-7564-46fe-8c50-efd4f07fa364"). InnerVolumeSpecName "kube-api-access-krgjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:50.654805 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.654771 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgjw\" (UniqueName: \"kubernetes.io/projected/19bd7a95-7564-46fe-8c50-efd4f07fa364-kube-api-access-krgjw\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:50.654805 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:50.654799 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bd7a95-7564-46fe-8c50-efd4f07fa364-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:25:51.119192 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.119151 5914 generic.go:356] "Generic (PLEG): container finished" podID="19bd7a95-7564-46fe-8c50-efd4f07fa364" containerID="b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef" exitCode=0 Oct 30 22:25:51.119700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.119208 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" event={"ID":"19bd7a95-7564-46fe-8c50-efd4f07fa364","Type":"ContainerDied","Data":"b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef"} Oct 30 22:25:51.119700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.119268 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" event={"ID":"19bd7a95-7564-46fe-8c50-efd4f07fa364","Type":"ContainerDied","Data":"c16cd87b95a97fd6b89790f4d3524ba00ffb97063d32d0bc2daf11c7093c40b8"} Oct 30 22:25:51.119700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.119193 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545" Oct 30 22:25:51.119700 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.119295 5914 scope.go:122] "RemoveContainer" containerID="b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef" Oct 30 22:25:51.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.120608 5914 generic.go:356] "Generic (PLEG): container finished" podID="e1a3428b-6271-472c-94ee-c4432dbb7c7d" containerID="2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55" exitCode=0 Oct 30 22:25:51.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.120654 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" Oct 30 22:25:51.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.120672 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" event={"ID":"e1a3428b-6271-472c-94ee-c4432dbb7c7d","Type":"ContainerDied","Data":"2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55"} Oct 30 22:25:51.120821 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.120705 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8" event={"ID":"e1a3428b-6271-472c-94ee-c4432dbb7c7d","Type":"ContainerDied","Data":"dd436f3ed3b20bceec42d473a0a72f0d95d18ece147cdfb56442dac1159fd40a"} Oct 30 22:25:51.130009 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.129863 5914 scope.go:122] "RemoveContainer" containerID="b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef" Oct 30 22:25:51.130243 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:25:51.130224 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef\": container with ID starting with b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef not found: ID does not exist" containerID="b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef" Oct 30 22:25:51.130305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.130254 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef"} err="failed to get container status \"b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef\": rpc error: code = NotFound desc = could not find container \"b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef\": container with ID starting with b8ed97ee48910a1dfcaf4e99e04a2d6e4ad38b61a46b2b44cfe3dda56e0d54ef not found: ID does not exist" Oct 30 22:25:51.130305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.130273 5914 scope.go:122] "RemoveContainer" containerID="2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55" Oct 30 22:25:51.139826 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.139802 5914 scope.go:122] "RemoveContainer" containerID="2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55" Oct 30 22:25:51.140118 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:25:51.140097 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55\": container with ID starting with 2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55 not found: ID does not exist" containerID="2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55" Oct 30 22:25:51.140178 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.140127 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55"} err="failed to get container status \"2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55\": rpc error: code = NotFound desc = could not find container \"2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55\": container with ID starting with 2854972c3cafa181c5af4b743dfb12d6afe9076199a2551a81982cdd6520bf55 not found: ID does not exist" Oct 30 22:25:51.281041 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.281009 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8"] Oct 30 22:25:51.330813 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.330728 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-daf0fd7e-5847b5cfb5-kdjv8"] Oct 30 22:25:51.394218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.394187 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545"] Oct 30 22:25:51.395666 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:51.395643 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-daf0fd7e-fc995d59-pc545"] Oct 30 22:25:52.215833 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:52.215801 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bd7a95-7564-46fe-8c50-efd4f07fa364" path="/var/lib/kubelet/pods/19bd7a95-7564-46fe-8c50-efd4f07fa364/volumes" Oct 30 22:25:52.216295 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:52.216279 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a3428b-6271-472c-94ee-c4432dbb7c7d" path="/var/lib/kubelet/pods/e1a3428b-6271-472c-94ee-c4432dbb7c7d/volumes" Oct 30 22:25:53.771850 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:53.771820 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:25:54.648121 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:54.648085 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr"] Oct 30 22:25:54.648985 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:54.648939 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" containerID="cri-o://15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f" gracePeriod=30 Oct 30 22:25:59.912891 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:25:59.912855 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:26:00.054866 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.054766 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.054866 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.054823 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.054866 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.054861 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.054957 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.054985 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055025 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055043 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055105 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055206 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055168 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token\") pod \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\" (UID: \"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e\") " Oct 30 22:26:00.055521 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055246 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:00.055521 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055274 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:00.055521 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055299 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:00.055521 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055309 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:00.055521 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055368 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:00.055762 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055592 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.055762 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055613 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.055762 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055628 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.055762 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055642 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.055762 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.055657 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.057669 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.057642 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:00.057669 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.057647 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "kube-api-access-cdvks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:00.057999 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.057982 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:00.058097 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.058071 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token" pod "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" (UID: "b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:00.156410 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.156370 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.156410 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.156404 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.156410 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.156415 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.156666 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.156424 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvks\" (UniqueName: \"kubernetes.io/projected/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e-kube-api-access-cdvks\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:26:00.171624 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.171589 5914 generic.go:356] "Generic (PLEG): container finished" podID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerID="15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f" exitCode=0 Oct 30 22:26:00.171798 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.171648 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" Oct 30 22:26:00.171798 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.171673 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" event={"ID":"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e","Type":"ContainerDied","Data":"15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f"} Oct 30 22:26:00.171798 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.171734 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr" event={"ID":"b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e","Type":"ContainerDied","Data":"fa5dba2302d2e6be96c29bf90eb64057c219d3ee781b017d52ce960e6095ad38"} Oct 30 22:26:00.171798 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.171756 5914 scope.go:122] "RemoveContainer" containerID="15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f" Oct 30 22:26:00.181937 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.181912 5914 scope.go:122] "RemoveContainer" containerID="5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a" Oct 30 22:26:00.190503 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.190481 5914 scope.go:122] "RemoveContainer" containerID="15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f" Oct 30 22:26:00.190782 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:26:00.190760 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f\": container with ID starting with 15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f not found: ID does not exist" containerID="15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f" Oct 30 22:26:00.190871 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.190791 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f"} err="failed to get container status \"15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f\": rpc error: code = NotFound desc = could not find container \"15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f\": container with ID starting with 15c8ac28d17c834fe0793d83c91b39a438a6125e822e18a80b00d565d031059f not found: ID does not exist" Oct 30 22:26:00.190871 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.190809 5914 scope.go:122] "RemoveContainer" containerID="5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a" Oct 30 22:26:00.191063 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:26:00.191045 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a\": container with ID starting with 5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a not found: ID does not exist" containerID="5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a" Oct 30 22:26:00.191102 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.191067 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a"} err="failed to get container status \"5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a\": rpc error: code = NotFound desc = could not find container \"5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a\": container with ID starting with 5ef6c0ad159a3b94cb2a971b6e492375c2f343ba8bec190fad7fe051fd26917a not found: ID does not exist" Oct 30 22:26:00.335567 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.335483 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr"] Oct 30 22:26:00.341112 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:00.341088 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-daf0fd7e-openshift-default-79b49d967-bl4pr"] Oct 30 22:26:02.216025 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:02.215985 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" path="/var/lib/kubelet/pods/b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e/volumes" Oct 30 22:26:06.468739 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.468700 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm"] Oct 30 22:26:06.469219 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469202 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e1a3428b-6271-472c-94ee-c4432dbb7c7d" containerName="payload-processing" Oct 30 22:26:06.469267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469221 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a3428b-6271-472c-94ee-c4432dbb7c7d" containerName="payload-processing" Oct 30 22:26:06.469267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469249 5914 cpu_manager.go:414] "containerMap: removing container" podUID="19bd7a95-7564-46fe-8c50-efd4f07fa364" containerName="payload-pre-processing" Oct 30 22:26:06.469267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469254 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bd7a95-7564-46fe-8c50-efd4f07fa364" containerName="payload-pre-processing" Oct 30 22:26:06.469267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469263 5914 cpu_manager.go:414] "containerMap: removing container" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" Oct 30 22:26:06.469267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469268 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" Oct 30 22:26:06.469439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469283 5914 cpu_manager.go:414] "containerMap: removing container" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" Oct 30 22:26:06.469439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469291 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" Oct 30 22:26:06.469439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469353 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a3428b-6271-472c-94ee-c4432dbb7c7d" containerName="payload-processing" Oct 30 22:26:06.469439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469369 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bd7a95-7564-46fe-8c50-efd4f07fa364" containerName="payload-pre-processing" Oct 30 22:26:06.469439 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.469377 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" Oct 30 22:26:06.472625 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.472604 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518249 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518217 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518249 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518257 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518498 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518297 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518498 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518399 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518498 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518473 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518530 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518622 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518566 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518690 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.518754 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.518714 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.524828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.524797 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-161f7a-openshift-default-dockercfg-4v8sv\"" Oct 30 22:26:06.577870 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.577723 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm"] Oct 30 22:26:06.619328 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619276 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619328 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619327 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619367 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619388 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619413 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619429 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619451 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619537 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.619586 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.619562 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.620305 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.620114 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.620433 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.620397 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.621228 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.621202 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.621717 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.621385 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.621717 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.621640 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.623419 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.623394 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.624423 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.624398 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.677771 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.677740 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.678361 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.678335 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token\") pod \"e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:06.785279 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:06.785190 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:07.218680 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:07.218613 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm"] Oct 30 22:26:07.221551 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:26:07.221525 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7677244_b1ae_47e1_8f76_c5773ff72e7f.slice/crio-91cfb3b368906ce8867d4c3bb7ac408491f7956b6327328d13ae4162e94e5ccb WatchSource:0}: Error finding container 91cfb3b368906ce8867d4c3bb7ac408491f7956b6327328d13ae4162e94e5ccb: Status 404 returned error can't find the container with id 91cfb3b368906ce8867d4c3bb7ac408491f7956b6327328d13ae4162e94e5ccb Oct 30 22:26:07.223635 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:07.223607 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:07.223711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:07.223664 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:07.223711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:07.223690 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:08.223829 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:08.223796 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" event={"ID":"c7677244-b1ae-47e1-8f76-c5773ff72e7f","Type":"ContainerStarted","Data":"06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9"} Oct 30 22:26:08.223829 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:08.223832 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" event={"ID":"c7677244-b1ae-47e1-8f76-c5773ff72e7f","Type":"ContainerStarted","Data":"91cfb3b368906ce8867d4c3bb7ac408491f7956b6327328d13ae4162e94e5ccb"} Oct 30 22:26:08.388708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:08.388646 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podStartSLOduration=2.388632625 podStartE2EDuration="2.388632625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:08.386780365 +0000 UTC m=+4120.288644443" watchObservedRunningTime="2026-10-30 22:26:08.388632625 +0000 UTC m=+4120.290496690" Oct 30 22:26:08.786062 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:08.786020 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:08.791304 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:08.791276 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:09.229096 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:09.229066 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:09.229989 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:09.229963 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:12.345862 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:26:12.345823 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:26:14.855125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.855082 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l"] Oct 30 22:26:14.855631 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.855618 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64b4dd5-a7a0-4d6a-ada9-4a97263b5b5e" containerName="istio-proxy" Oct 30 22:26:14.859055 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.859034 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:14.905744 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.905701 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9zh\" (UniqueName: \"kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh\") pod \"payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:14.905928 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.905762 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:14.921127 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.921091 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l"] Oct 30 22:26:14.967356 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:14.967322 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-161f7a\"" Oct 30 22:26:15.007501 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.007453 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9zh\" (UniqueName: \"kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh\") pod \"payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:15.007820 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.007527 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:15.008530 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.008504 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:15.008629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.008608 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm"] Oct 30 22:26:15.012462 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.012441 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.065660 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.065622 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9zh\" (UniqueName: \"kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh\") pod \"payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:15.068108 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.068070 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-161f7a-dockercfg-j8bvj\"" Oct 30 22:26:15.068673 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.068650 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm"] Oct 30 22:26:15.108790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.108705 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.108790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.108765 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr\") pod \"payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.172667 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.172639 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:15.209447 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.209408 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.209629 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.209503 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr\") pod \"payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.210150 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.210127 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.266568 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.266537 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr\") pod \"payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.325839 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.325809 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:15.550008 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.549977 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l"] Oct 30 22:26:15.551766 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:26:15.551737 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbfc44a_3ed5_47d1_ab76_e31403d0078f.slice/crio-d196c6db9da2bbd1e322eba334ba3c9faef606b55770014b163c7cff6c7f7887 WatchSource:0}: Error finding container d196c6db9da2bbd1e322eba334ba3c9faef606b55770014b163c7cff6c7f7887: Status 404 returned error can't find the container with id d196c6db9da2bbd1e322eba334ba3c9faef606b55770014b163c7cff6c7f7887 Oct 30 22:26:15.698946 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:15.698902 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm"] Oct 30 22:26:15.701511 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:26:15.701481 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6de86c7_674c_4249_858b_a34878983881.slice/crio-c643d244456876750f2f4484e3773ca07b53f28986c5f396c7fc4ac7a8654e3e WatchSource:0}: Error finding container c643d244456876750f2f4484e3773ca07b53f28986c5f396c7fc4ac7a8654e3e: Status 404 returned error can't find the container with id c643d244456876750f2f4484e3773ca07b53f28986c5f396c7fc4ac7a8654e3e Oct 30 22:26:16.266591 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.266559 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" event={"ID":"6cbfc44a-3ed5-47d1-ab76-e31403d0078f","Type":"ContainerStarted","Data":"42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3"} Oct 30 22:26:16.267195 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.266610 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" event={"ID":"6cbfc44a-3ed5-47d1-ab76-e31403d0078f","Type":"ContainerStarted","Data":"d196c6db9da2bbd1e322eba334ba3c9faef606b55770014b163c7cff6c7f7887"} Oct 30 22:26:16.267195 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.266648 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:16.268309 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.268287 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" event={"ID":"e6de86c7-674c-4249-858b-a34878983881","Type":"ContainerStarted","Data":"9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a"} Oct 30 22:26:16.268428 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.268324 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" event={"ID":"e6de86c7-674c-4249-858b-a34878983881","Type":"ContainerStarted","Data":"c643d244456876750f2f4484e3773ca07b53f28986c5f396c7fc4ac7a8654e3e"} Oct 30 22:26:16.268428 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.268365 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:16.381690 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.381640 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" podStartSLOduration=2.381622542 podStartE2EDuration="2.381622542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:16.380336537 +0000 UTC m=+4128.282200618" watchObservedRunningTime="2026-10-30 22:26:16.381622542 +0000 UTC m=+4128.283486607" Oct 30 22:26:16.497669 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:16.495638 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" podStartSLOduration=2.495617577 podStartE2EDuration="2.495617577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:16.49421437 +0000 UTC m=+4128.396078435" watchObservedRunningTime="2026-10-30 22:26:16.495617577 +0000 UTC m=+4128.397481646" Oct 30 22:26:25.206354 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:25.206317 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:26:27.299110 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:27.299075 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:26:27.299933 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:27.299913 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:26:35.373267 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:35.373236 5914 generic.go:356] "Generic (PLEG): container finished" podID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerID="06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9" exitCode=0 Oct 30 22:26:35.373946 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:35.373289 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" event={"ID":"c7677244-b1ae-47e1-8f76-c5773ff72e7f","Type":"ContainerDied","Data":"06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9"} Oct 30 22:26:35.373946 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:35.373650 5914 scope.go:122] "RemoveContainer" containerID="06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9" Oct 30 22:26:35.382413 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:35.375598 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:35.382413 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:35.375770 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:35.382413 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:35.375869 5914 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:36.403135 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:36.403040 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" event={"ID":"c7677244-b1ae-47e1-8f76-c5773ff72e7f","Type":"ContainerStarted","Data":"608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb"} Oct 30 22:26:36.793968 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:36.793269 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:36.797906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:36.795902 5914 patch_prober.go:29] interesting pod/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:26:36.797906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:36.795967 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 22:26:37.786405 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:37.786367 5914 patch_prober.go:29] interesting pod/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:26:37.787033 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:37.787000 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 22:26:38.791372 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:38.791335 5914 patch_prober.go:29] interesting pod/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:26:38.792041 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:38.792007 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 22:26:39.229551 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:39.229517 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:39.796906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:39.796362 5914 patch_prober.go:29] interesting pod/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:26:39.796906 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:39.796431 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 22:26:40.792907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:40.787602 5914 patch_prober.go:29] interesting pod/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:26:40.792907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:40.787678 5914 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 30 22:26:41.790324 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:41.790291 5914 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:41.791380 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:26:41.791357 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:26:42.345219 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:26:42.345183 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:27:12.345377 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:27:12.345336 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:27:29.513531 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:27:29.513498 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:27:29.531002 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:27:29.530971 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:27:42.345428 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:27:42.345386 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:28:12.350143 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:28:12.350057 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:28:28.610277 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:28.610244 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm"] Oct 30 22:28:28.718695 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:28.718658 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l"] Oct 30 22:28:33.610786 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.610724 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" podUID="e6de86c7-674c-4249-858b-a34878983881" containerName="payload-processing" containerID="cri-o://9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a" gracePeriod=25 Oct 30 22:28:33.719501 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.719449 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" podUID="6cbfc44a-3ed5-47d1-ab76-e31403d0078f" containerName="payload-pre-processing" containerID="cri-o://42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3" gracePeriod=25 Oct 30 22:28:33.885926 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.885687 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:28:33.976348 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.976316 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:28:33.992649 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.992621 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr\") pod \"e6de86c7-674c-4249-858b-a34878983881\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " Oct 30 22:28:33.992817 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.992799 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume\") pod \"e6de86c7-674c-4249-858b-a34878983881\" (UID: \"e6de86c7-674c-4249-858b-a34878983881\") " Oct 30 22:28:33.993247 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.993218 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume" pod "e6de86c7-674c-4249-858b-a34878983881" (UID: "e6de86c7-674c-4249-858b-a34878983881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:33.994856 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:33.994829 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr" pod "e6de86c7-674c-4249-858b-a34878983881" (UID: "e6de86c7-674c-4249-858b-a34878983881"). InnerVolumeSpecName "kube-api-access-t5cnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:34.005581 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.005546 5914 generic.go:356] "Generic (PLEG): container finished" podID="6cbfc44a-3ed5-47d1-ab76-e31403d0078f" containerID="42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3" exitCode=0 Oct 30 22:28:34.005758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.005595 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" Oct 30 22:28:34.005758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.005622 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" event={"ID":"6cbfc44a-3ed5-47d1-ab76-e31403d0078f","Type":"ContainerDied","Data":"42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3"} Oct 30 22:28:34.005758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.005678 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l" event={"ID":"6cbfc44a-3ed5-47d1-ab76-e31403d0078f","Type":"ContainerDied","Data":"d196c6db9da2bbd1e322eba334ba3c9faef606b55770014b163c7cff6c7f7887"} Oct 30 22:28:34.005758 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.005698 5914 scope.go:122] "RemoveContainer" containerID="42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3" Oct 30 22:28:34.007122 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.007098 5914 generic.go:356] "Generic (PLEG): container finished" podID="e6de86c7-674c-4249-858b-a34878983881" containerID="9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a" exitCode=0 Oct 30 22:28:34.007246 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.007136 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" event={"ID":"e6de86c7-674c-4249-858b-a34878983881","Type":"ContainerDied","Data":"9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a"} Oct 30 22:28:34.007246 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.007156 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" Oct 30 22:28:34.007246 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.007172 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm" event={"ID":"e6de86c7-674c-4249-858b-a34878983881","Type":"ContainerDied","Data":"c643d244456876750f2f4484e3773ca07b53f28986c5f396c7fc4ac7a8654e3e"} Oct 30 22:28:34.019752 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.019726 5914 scope.go:122] "RemoveContainer" containerID="42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3" Oct 30 22:28:34.020132 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:28:34.020110 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3\": container with ID starting with 42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3 not found: ID does not exist" containerID="42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3" Oct 30 22:28:34.020243 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.020148 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3"} err="failed to get container status \"42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3\": rpc error: code = NotFound desc = could not find container \"42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3\": container with ID starting with 42bfe134d7f15d87143aafb0cbc8c3624260f7b20b5251d2613ee6818c50ffb3 not found: ID does not exist" Oct 30 22:28:34.020243 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.020172 5914 scope.go:122] "RemoveContainer" containerID="9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a" Oct 30 22:28:34.031818 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.031793 5914 scope.go:122] "RemoveContainer" containerID="9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a" Oct 30 22:28:34.032235 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:28:34.032211 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a\": container with ID starting with 9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a not found: ID does not exist" containerID="9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a" Oct 30 22:28:34.032314 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.032245 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a"} err="failed to get container status \"9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a\": rpc error: code = NotFound desc = could not find container \"9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a\": container with ID starting with 9629872e18f10d00b52e11e23a575d47438c71c22c943eaf80434d21362e8f4a not found: ID does not exist" Oct 30 22:28:34.093777 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.093742 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume\") pod \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " Oct 30 22:28:34.094017 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.093816 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh\" (UniqueName: \"kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh\") pod \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\" (UID: \"6cbfc44a-3ed5-47d1-ab76-e31403d0078f\") " Oct 30 22:28:34.094160 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.094135 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume" pod "6cbfc44a-3ed5-47d1-ab76-e31403d0078f" (UID: "6cbfc44a-3ed5-47d1-ab76-e31403d0078f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:34.094240 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.094220 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6de86c7-674c-4249-858b-a34878983881-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.094291 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.094248 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5cnr\" (UniqueName: \"kubernetes.io/projected/e6de86c7-674c-4249-858b-a34878983881-kube-api-access-t5cnr\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.096029 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.096008 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh" pod "6cbfc44a-3ed5-47d1-ab76-e31403d0078f" (UID: "6cbfc44a-3ed5-47d1-ab76-e31403d0078f"). InnerVolumeSpecName "kube-api-access-2q9zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:34.195074 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.194990 5914 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-plugins-config-volume\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.195074 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.195019 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q9zh\" (UniqueName: \"kubernetes.io/projected/6cbfc44a-3ed5-47d1-ab76-e31403d0078f-kube-api-access-2q9zh\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:34.271813 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.271778 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm"] Oct 30 22:28:34.275271 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.275244 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-161f7a-5674f96d7f-rv7dm"] Oct 30 22:28:34.419150 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.419114 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l"] Oct 30 22:28:34.422468 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:34.422438 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-161f7a-57b7db64ff-fcc9l"] Oct 30 22:28:36.217311 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:36.217273 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbfc44a-3ed5-47d1-ab76-e31403d0078f" path="/var/lib/kubelet/pods/6cbfc44a-3ed5-47d1-ab76-e31403d0078f/volumes" Oct 30 22:28:36.217875 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:36.217852 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6de86c7-674c-4249-858b-a34878983881" path="/var/lib/kubelet/pods/e6de86c7-674c-4249-858b-a34878983881/volumes" Oct 30 22:28:41.268925 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:41.268871 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:28:42.345342 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:28:42.345304 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:28:45.135596 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:45.135561 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm"] Oct 30 22:28:45.136085 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:45.135780 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" containerID="cri-o://608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb" gracePeriod=30 Oct 30 22:28:50.377957 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.377928 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:28:50.464194 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464120 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464340 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464222 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464340 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464255 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464340 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464282 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464445 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464419 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:50.464513 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464487 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464601 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464583 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464669 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464649 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464734 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464679 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464756 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs\") pod \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\" (UID: \"c7677244-b1ae-47e1-8f76-c5773ff72e7f\") " Oct 30 22:28:50.464797 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.464758 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:50.465114 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465089 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:50.465234 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465125 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:50.465342 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465321 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.465398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465348 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.465398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465363 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.465398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465379 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.465574 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.465396 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:50.466662 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.466632 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:50.466775 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.466659 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:50.466775 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.466712 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:50.467117 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.467102 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb" pod "c7677244-b1ae-47e1-8f76-c5773ff72e7f" (UID: "c7677244-b1ae-47e1-8f76-c5773ff72e7f"). InnerVolumeSpecName "kube-api-access-xbsvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:50.566296 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.566262 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.566296 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.566295 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-kube-api-access-xbsvb\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.566510 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.566311 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.566510 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.566324 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:50.566510 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:50.566336 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7677244-b1ae-47e1-8f76-c5773ff72e7f-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:28:51.104968 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.104936 5914 generic.go:356] "Generic (PLEG): container finished" podID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerID="608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb" exitCode=0 Oct 30 22:28:51.105125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.104985 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" event={"ID":"c7677244-b1ae-47e1-8f76-c5773ff72e7f","Type":"ContainerDied","Data":"608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb"} Oct 30 22:28:51.105125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.105026 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" Oct 30 22:28:51.105125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.105040 5914 scope.go:122] "RemoveContainer" containerID="608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb" Oct 30 22:28:51.105125 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.105031 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm" event={"ID":"c7677244-b1ae-47e1-8f76-c5773ff72e7f","Type":"ContainerDied","Data":"91cfb3b368906ce8867d4c3bb7ac408491f7956b6327328d13ae4162e94e5ccb"} Oct 30 22:28:51.115450 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.115431 5914 scope.go:122] "RemoveContainer" containerID="06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9" Oct 30 22:28:51.123855 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.123833 5914 scope.go:122] "RemoveContainer" containerID="608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb" Oct 30 22:28:51.124154 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:28:51.124135 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb\": container with ID starting with 608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb not found: ID does not exist" containerID="608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb" Oct 30 22:28:51.124208 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.124164 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb"} err="failed to get container status \"608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb\": rpc error: code = NotFound desc = could not find container \"608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb\": container with ID starting with 608c765e036b6bb9c3764b63684295630103a2953b474599b5b57ca98a6f4abb not found: ID does not exist" Oct 30 22:28:51.124208 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.124183 5914 scope.go:122] "RemoveContainer" containerID="06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9" Oct 30 22:28:51.124413 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:28:51.124395 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9\": container with ID starting with 06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9 not found: ID does not exist" containerID="06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9" Oct 30 22:28:51.124455 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.124420 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9"} err="failed to get container status \"06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9\": rpc error: code = NotFound desc = could not find container \"06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9\": container with ID starting with 06b4ecfef29196ef7da1cea06547941e3c7a77069ab729f1a4709456b85417e9 not found: ID does not exist" Oct 30 22:28:51.215106 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.215071 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:28:51.330286 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.330256 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm"] Oct 30 22:28:51.343795 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:51.343766 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-161f7a-openshift-default-5f8b8f6797-ph5qm"] Oct 30 22:28:52.219910 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:28:52.219844 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" path="/var/lib/kubelet/pods/c7677244-b1ae-47e1-8f76-c5773ff72e7f/volumes" Oct 30 22:29:04.920379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:29:04.920341 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:29:12.344712 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:29:12.344666 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:29:25.908846 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:29:25.908804 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:29:42.345133 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:29:42.345052 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:29:58.604116 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:29:58.604073 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-5lhdg"] Oct 30 22:30:12.345105 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:30:12.345066 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:30:29.454874 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:30:29.454843 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rmhhv_957309ba-a518-4e31-b235-850d1d0a87bd/manager/0.log" Oct 30 22:30:30.389100 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:30:30.389071 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58d6bcb5f-4h2fw_3de5dbd8-6c4e-480c-9f2b-4236665c3dcc/manager/0.log" Oct 30 22:30:31.307209 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:30:31.307169 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-25psm_ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2/manager/0.log" Oct 30 22:30:32.462709 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:30:32.462680 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5644488b47-cvjsn_d3c285e1-7da5-4065-a3dc-443854d0bf10/maas-api/0.log" Oct 30 22:30:42.345158 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:30:42.345116 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:30:43.680482 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:30:43.680455 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-xgzdn_78568b14-83eb-409f-ac90-5908ca352923/postgres/0.log" Oct 30 22:31:06.518327 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:06.518238 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4xdrt_55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b/manager/0.log" Oct 30 22:31:07.107167 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:07.107134 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5lhdg_0a535427-8427-4cc8-a23b-030dcb993336/limitador/0.log" Oct 30 22:31:08.848131 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:08.848098 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-578b8bf98-4lglr_9b32414f-a7bf-4d68-bdc7-6568dc26d900/kube-auth-proxy/0.log" Oct 30 22:31:09.300664 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:09.300633 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-66hx9_bee23856-34e6-4110-8e8f-ae9d13253bcf/istio-proxy/0.log" Oct 30 22:31:10.373379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:10.373348 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-lwsc9_dce3fca9-5560-447a-9934-6a571dc6367a/istio-proxy/0.log" Oct 30 22:31:12.344607 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:31:12.344549 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:31:20.611191 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:20.611160 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mblkw_f9a11a6a-ccd2-414b-a170-d066c51576b1/istio-proxy/0.log" Oct 30 22:31:21.686540 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:21.686508 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-zqcqx_845ebdc3-490d-4413-bab1-6838ea3d3603/istio-proxy/0.log" Oct 30 22:31:22.441790 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:22.441752 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fgz7d_22e2cb45-b7df-4a03-8a39-883b748adfb5/payload-processing/0.log" Oct 30 22:31:23.605334 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:23.605303 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-p878m_58a9ebae-7b84-4141-8410-4b0b9225c81f/main/0.log" Oct 30 22:31:24.107651 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:24.107618 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kg4x4_0902fa42-613e-4b60-a4f5-c6e0b0857801/main/0.log" Oct 30 22:31:24.333896 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:24.333858 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8mp4b_0e3715d9-a970-425c-88df-4f3544ea45d9/main/0.log" Oct 30 22:31:24.803236 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:31:24.803208 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4qch8_10913186-88c2-40e0-ba50-a2c7f26c64ad/main/0.log" Oct 30 22:31:42.344767 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:31:42.344730 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:32:05.973126 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:05.973096 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rmhhv_957309ba-a518-4e31-b235-850d1d0a87bd/manager/0.log" Oct 30 22:32:06.864824 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:06.864742 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58d6bcb5f-4h2fw_3de5dbd8-6c4e-480c-9f2b-4236665c3dcc/manager/0.log" Oct 30 22:32:07.492830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:07.492797 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-25psm_ad4f2d63-5bcf-4303-93b8-7279ba0f7cf2/manager/0.log" Oct 30 22:32:12.344730 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:32:12.344692 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:32:18.333709 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:18.333678 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5644488b47-cvjsn_d3c285e1-7da5-4065-a3dc-443854d0bf10/maas-api/0.log" Oct 30 22:32:19.645180 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:19.645148 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-xgzdn_78568b14-83eb-409f-ac90-5908ca352923/postgres/0.log" Oct 30 22:32:29.545827 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:29.545799 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:32:29.565332 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:29.565304 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:32:42.150399 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:42.150309 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4xdrt_55c17f4d-2ea4-4bf3-a9a2-1e2859f9288b/manager/0.log" Oct 30 22:32:42.345346 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:32:42.345308 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:32:42.617133 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:42.617046 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-5lhdg_0a535427-8427-4cc8-a23b-030dcb993336/limitador/0.log" Oct 30 22:32:44.098851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:44.098822 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-578b8bf98-4lglr_9b32414f-a7bf-4d68-bdc7-6568dc26d900/kube-auth-proxy/0.log" Oct 30 22:32:44.550450 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:44.550416 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-66hx9_bee23856-34e6-4110-8e8f-ae9d13253bcf/istio-proxy/0.log" Oct 30 22:32:45.607504 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:45.607471 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-lwsc9_dce3fca9-5560-447a-9934-6a571dc6367a/istio-proxy/0.log" Oct 30 22:32:45.840218 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:45.840186 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mblkw_f9a11a6a-ccd2-414b-a170-d066c51576b1/istio-proxy/0.log" Oct 30 22:32:56.900398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:56.900367 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-zqcqx_845ebdc3-490d-4413-bab1-6838ea3d3603/istio-proxy/0.log" Oct 30 22:32:57.637363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:32:57.637337 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fgz7d_22e2cb45-b7df-4a03-8a39-883b748adfb5/payload-processing/0.log" Oct 30 22:33:08.548056 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:33:08.548017 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-p878m_58a9ebae-7b84-4141-8410-4b0b9225c81f/main/0.log" Oct 30 22:33:12.346544 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:33:12.346486 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:33:19.077184 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:33:19.077151 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kg4x4_0902fa42-613e-4b60-a4f5-c6e0b0857801/main/0.log" Oct 30 22:33:19.366190 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:33:19.366109 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8mp4b_0e3715d9-a970-425c-88df-4f3544ea45d9/main/0.log" Oct 30 22:33:29.878136 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:33:29.878086 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4qch8_10913186-88c2-40e0-ba50-a2c7f26c64ad/main/0.log" Oct 30 22:33:42.345713 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:33:42.345676 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:34:12.345124 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:34:12.345042 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:34:42.345188 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:34:42.345144 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:35:12.345587 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:35:12.345541 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:35:15.786680 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:15.786631 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx"] Oct 30 22:35:15.787177 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:15.786856 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" containerID="cri-o://9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8" gracePeriod=30 Oct 30 22:35:20.670927 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:20.670873 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58d6bcb5f-4h2fw_3de5dbd8-6c4e-480c-9f2b-4236665c3dcc/manager/0.log" Oct 30 22:35:21.048908 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.048868 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:35:21.115159 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115125 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115185 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115216 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115248 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115302 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115363 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115335 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115659 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115372 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115659 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115376 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:21.115659 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115420 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115659 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115493 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token\") pod \"845ebdc3-490d-4413-bab1-6838ea3d3603\" (UID: \"845ebdc3-490d-4413-bab1-6838ea3d3603\") " Oct 30 22:35:21.115659 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115557 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:21.115982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115695 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:21.115982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115781 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:21.115982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115910 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.115982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115932 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.115982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115946 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.115982 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.115962 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.116483 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.116006 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:21.117778 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.117747 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:21.117924 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.117787 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:21.117924 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.117836 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:21.117924 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.117846 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp" pod "845ebdc3-490d-4413-bab1-6838ea3d3603" (UID: "845ebdc3-490d-4413-bab1-6838ea3d3603"). InnerVolumeSpecName "kube-api-access-mzkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:21.134583 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.134555 5914 generic.go:356] "Generic (PLEG): container finished" podID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerID="9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8" exitCode=0 Oct 30 22:35:21.134711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.134595 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" Oct 30 22:35:21.134711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.134630 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" event={"ID":"845ebdc3-490d-4413-bab1-6838ea3d3603","Type":"ContainerDied","Data":"9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8"} Oct 30 22:35:21.134711 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.134688 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx" event={"ID":"845ebdc3-490d-4413-bab1-6838ea3d3603","Type":"ContainerDied","Data":"856ceecfaa34ffd37465c5cfdb6e6e7f5e2492f83a71423aa9605d9106bcccbf"} Oct 30 22:35:21.134818 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.134717 5914 scope.go:122] "RemoveContainer" containerID="9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8" Oct 30 22:35:21.144193 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.144175 5914 scope.go:122] "RemoveContainer" containerID="9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8" Oct 30 22:35:21.144455 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:35:21.144433 5914 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8\": container with ID starting with 9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8 not found: ID does not exist" containerID="9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8" Oct 30 22:35:21.144502 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.144465 5914 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8"} err="failed to get container status \"9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8\": rpc error: code = NotFound desc = could not find container \"9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8\": container with ID starting with 9e35ce2fd0175ffdacb04dd8ac08c1e095c0c6ce67467b762e10993f436341c8 not found: ID does not exist" Oct 30 22:35:21.216828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.216747 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzkkp\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-kube-api-access-mzkkp\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.216828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.216773 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/845ebdc3-490d-4413-bab1-6838ea3d3603-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.216828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.216784 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.216828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.216793 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.216828 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.216801 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/845ebdc3-490d-4413-bab1-6838ea3d3603-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.343301 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.343266 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx"] Oct 30 22:35:21.352673 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.352637 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-zqcqx"] Oct 30 22:35:21.442524 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:21.442496 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58d6bcb5f-4h2fw_3de5dbd8-6c4e-480c-9f2b-4236665c3dcc/manager/0.log" Oct 30 22:35:22.216243 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:22.216212 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" path="/var/lib/kubelet/pods/845ebdc3-490d-4413-bab1-6838ea3d3603/volumes" Oct 30 22:35:22.257175 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:22.257149 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fgz7d_22e2cb45-b7df-4a03-8a39-883b748adfb5/payload-processing/0.log" Oct 30 22:35:22.499045 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:22.498968 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fgz7d_22e2cb45-b7df-4a03-8a39-883b748adfb5/payload-processing/0.log" Oct 30 22:35:30.901398 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:30.901360 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9"] Oct 30 22:35:30.901858 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:30.901636 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" containerID="cri-o://b93fab8d55833bf9b1321afc803d623ee9e9f9145ee89877d3b46ab00fe27b0f" gracePeriod=30 Oct 30 22:35:30.950937 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:30.950898 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw"] Oct 30 22:35:30.951164 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:30.951124 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" containerID="cri-o://aa2f947c690309b80d978a1c1f685b64dcdf1dceddf6399ee2dc2768b6622fd9" gracePeriod=30 Oct 30 22:35:30.952016 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:30.951993 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9"] Oct 30 22:35:30.952221 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:30.952177 5914 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" containerID="cri-o://d878b24f4556539150ba57fc60b2fc69019dfa680c35256f676fe30cd4f6001b" gracePeriod=30 Oct 30 22:35:36.209158 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.209120 5914 generic.go:356] "Generic (PLEG): container finished" podID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerID="d878b24f4556539150ba57fc60b2fc69019dfa680c35256f676fe30cd4f6001b" exitCode=0 Oct 30 22:35:36.211994 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.211917 5914 generic.go:356] "Generic (PLEG): container finished" podID="dce3fca9-5560-447a-9934-6a571dc6367a" containerID="b93fab8d55833bf9b1321afc803d623ee9e9f9145ee89877d3b46ab00fe27b0f" exitCode=0 Oct 30 22:35:36.214644 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.214622 5914 generic.go:356] "Generic (PLEG): container finished" podID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerID="aa2f947c690309b80d978a1c1f685b64dcdf1dceddf6399ee2dc2768b6622fd9" exitCode=0 Oct 30 22:35:36.217667 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.217644 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" event={"ID":"bee23856-34e6-4110-8e8f-ae9d13253bcf","Type":"ContainerDied","Data":"d878b24f4556539150ba57fc60b2fc69019dfa680c35256f676fe30cd4f6001b"} Oct 30 22:35:36.217768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.217684 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" event={"ID":"dce3fca9-5560-447a-9934-6a571dc6367a","Type":"ContainerDied","Data":"b93fab8d55833bf9b1321afc803d623ee9e9f9145ee89877d3b46ab00fe27b0f"} Oct 30 22:35:36.217768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.217703 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" event={"ID":"dce3fca9-5560-447a-9934-6a571dc6367a","Type":"ContainerDied","Data":"55bc0de7a4439f45a559a5f2e1072685739b2ff8fa800f94d9bd35bfa8cb6373"} Oct 30 22:35:36.217768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.217716 5914 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bc0de7a4439f45a559a5f2e1072685739b2ff8fa800f94d9bd35bfa8cb6373" Oct 30 22:35:36.217768 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.217724 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" event={"ID":"f9a11a6a-ccd2-414b-a170-d066c51576b1","Type":"ContainerDied","Data":"aa2f947c690309b80d978a1c1f685b64dcdf1dceddf6399ee2dc2768b6622fd9"} Oct 30 22:35:36.223178 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.223162 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:35:36.249262 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.249236 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:35:36.252503 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.252483 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.368865 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.368943 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.368985 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369012 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369039 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369062 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369091 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369122 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369151 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369180 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369211 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369234 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369251 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369290 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369317 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369337 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369358 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369378 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369432 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369460 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369483 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369504 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369547 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy\") pod \"bee23856-34e6-4110-8e8f-ae9d13253bcf\" (UID: \"bee23856-34e6-4110-8e8f-ae9d13253bcf\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369578 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369606 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs\") pod \"f9a11a6a-ccd2-414b-a170-d066c51576b1\" (UID: \"f9a11a6a-ccd2-414b-a170-d066c51576b1\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369632 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.369667 5914 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs\") pod \"dce3fca9-5560-447a-9934-6a571dc6367a\" (UID: \"dce3fca9-5560-447a-9934-6a571dc6367a\") " Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.370468 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.370703 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.370793 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.370723 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.372830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.371281 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.372830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.371444 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.372830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.372446 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.372830 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.372631 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.373477 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373288 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.373477 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373325 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:36.373477 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373330 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:36.373852 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373621 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.373852 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373642 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.373852 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373710 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "kube-api-access-27lcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.374119 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.373998 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:36.374299 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.374272 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.374380 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.374275 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.374784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.374748 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.374784 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.374772 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.375823 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.375788 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:36.376528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.376442 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.376528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.376476 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.376528 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.376522 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:36.376848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.376588 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:36.376848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.376588 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:36.376848 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.376665 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx" pod "dce3fca9-5560-447a-9934-6a571dc6367a" (UID: "dce3fca9-5560-447a-9934-6a571dc6367a"). InnerVolumeSpecName "kube-api-access-6hckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.377183 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.377166 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token" pod "f9a11a6a-ccd2-414b-a170-d066c51576b1" (UID: "f9a11a6a-ccd2-414b-a170-d066c51576b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.378053 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.378034 5914 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj" pod "bee23856-34e6-4110-8e8f-ae9d13253bcf" (UID: "bee23856-34e6-4110-8e8f-ae9d13253bcf"). InnerVolumeSpecName "kube-api-access-lbqdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470846 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470905 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470916 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470925 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470934 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470943 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hckx\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-kube-api-access-6hckx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470954 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470962 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.470967 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470971 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470979 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqdj\" (UniqueName: \"kubernetes.io/projected/bee23856-34e6-4110-8e8f-ae9d13253bcf-kube-api-access-lbqdj\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470988 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bee23856-34e6-4110-8e8f-ae9d13253bcf-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.470996 5914 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-credential-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471004 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a11a6a-ccd2-414b-a170-d066c51576b1-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471012 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471020 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471027 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dce3fca9-5560-447a-9934-6a571dc6367a-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471038 5914 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27lcx\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-kube-api-access-27lcx\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471046 5914 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-data\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471054 5914 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-workload-socket\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471062 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471071 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471078 5914 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a11a6a-ccd2-414b-a170-d066c51576b1-istio-token\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471086 5914 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dce3fca9-5560-447a-9934-6a571dc6367a-istio-podinfo\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471094 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bee23856-34e6-4110-8e8f-ae9d13253bcf-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471101 5914 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dce3fca9-5560-447a-9934-6a571dc6367a-istio-envoy\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471109 5914 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a11a6a-ccd2-414b-a170-d066c51576b1-workload-certs\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:36.471379 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:36.471116 5914 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dce3fca9-5560-447a-9934-6a571dc6367a-istiod-ca-cert\") on node \"ip-10-0-142-119.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:35:37.221452 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.221349 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" Oct 30 22:35:37.221452 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.221361 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9" event={"ID":"bee23856-34e6-4110-8e8f-ae9d13253bcf","Type":"ContainerDied","Data":"759cd932809be06836199d26a148a198e7ba36466a77d94188da86ec2f33e84b"} Oct 30 22:35:37.221452 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.221438 5914 scope.go:122] "RemoveContainer" containerID="d878b24f4556539150ba57fc60b2fc69019dfa680c35256f676fe30cd4f6001b" Oct 30 22:35:37.222959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.222806 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9" Oct 30 22:35:37.222959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.222810 5914 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" Oct 30 22:35:37.222959 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.222811 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw" event={"ID":"f9a11a6a-ccd2-414b-a170-d066c51576b1","Type":"ContainerDied","Data":"2efd636207075dd59deef45984f832dad378a9c6e0ab9700591585308489bff3"} Oct 30 22:35:37.231264 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.231235 5914 scope.go:122] "RemoveContainer" containerID="aa2f947c690309b80d978a1c1f685b64dcdf1dceddf6399ee2dc2768b6622fd9" Oct 30 22:35:37.437062 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.437020 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9"] Oct 30 22:35:37.487070 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.486979 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-lwsc9"] Oct 30 22:35:37.553693 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.553658 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9"] Oct 30 22:35:37.557751 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.557714 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-66hx9"] Oct 30 22:35:37.667191 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.667157 5914 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw"] Oct 30 22:35:37.670378 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:37.670348 5914 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mblkw"] Oct 30 22:35:38.216345 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:38.216312 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" path="/var/lib/kubelet/pods/bee23856-34e6-4110-8e8f-ae9d13253bcf/volumes" Oct 30 22:35:38.216746 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:38.216733 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" path="/var/lib/kubelet/pods/dce3fca9-5560-447a-9934-6a571dc6367a/volumes" Oct 30 22:35:38.217154 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:35:38.217141 5914 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" path="/var/lib/kubelet/pods/f9a11a6a-ccd2-414b-a170-d066c51576b1/volumes" Oct 30 22:35:42.345552 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:35:42.345516 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:36:12.344996 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:36:12.344952 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:36:30.623984 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:36:30.623952 5914 scope.go:122] "RemoveContainer" containerID="b93fab8d55833bf9b1321afc803d623ee9e9f9145ee89877d3b46ab00fe27b0f" Oct 30 22:36:42.344665 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:36:42.344627 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:37:12.345282 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:37:12.345198 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:37:21.976834 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:21.976798 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58d6bcb5f-4h2fw_3de5dbd8-6c4e-480c-9f2b-4236665c3dcc/manager/0.log" Oct 30 22:37:22.465820 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:22.465786 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58d6bcb5f-4h2fw_3de5dbd8-6c4e-480c-9f2b-4236665c3dcc/manager/0.log" Oct 30 22:37:23.238032 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:23.238004 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fgz7d_22e2cb45-b7df-4a03-8a39-883b748adfb5/payload-processing/0.log" Oct 30 22:37:29.577090 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:29.577061 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:37:29.597254 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:29.597229 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-69lqx_4229762d-c3ea-43fd-851d-506bba6c76db/ovn-acl-logging/0.log" Oct 30 22:37:33.460827 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.460796 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fgz7d_22e2cb45-b7df-4a03-8a39-883b748adfb5/payload-processing/0.log" Oct 30 22:37:33.837067 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.836979 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tlnnt/must-gather-f9ddm"] Oct 30 22:37:33.837500 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837480 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837501 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837517 5914 cpu_manager.go:414] "containerMap: removing container" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837523 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837541 5914 cpu_manager.go:414] "containerMap: removing container" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837548 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837571 5914 cpu_manager.go:414] "containerMap: removing container" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837577 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837590 5914 cpu_manager.go:414] "containerMap: removing container" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" Oct 30 22:37:33.837603 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837598 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837613 5914 cpu_manager.go:414] "containerMap: removing container" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837620 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837632 5914 cpu_manager.go:414] "containerMap: removing container" podUID="6cbfc44a-3ed5-47d1-ab76-e31403d0078f" containerName="payload-pre-processing" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837640 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbfc44a-3ed5-47d1-ab76-e31403d0078f" containerName="payload-pre-processing" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837691 5914 cpu_manager.go:414] "containerMap: removing container" podUID="e6de86c7-674c-4249-858b-a34878983881" containerName="payload-processing" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837701 5914 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6de86c7-674c-4249-858b-a34878983881" containerName="payload-processing" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837808 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6de86c7-674c-4249-858b-a34878983881" containerName="payload-processing" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837825 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837835 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee23856-34e6-4110-8e8f-ae9d13253bcf" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837845 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a11a6a-ccd2-414b-a170-d066c51576b1" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837855 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="845ebdc3-490d-4413-bab1-6838ea3d3603" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837864 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce3fca9-5560-447a-9934-6a571dc6367a" containerName="istio-proxy" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.837872 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbfc44a-3ed5-47d1-ab76-e31403d0078f" containerName="payload-pre-processing" Oct 30 22:37:33.838118 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.838106 5914 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7677244-b1ae-47e1-8f76-c5773ff72e7f" containerName="istio-proxy" Oct 30 22:37:33.841372 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.841350 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:33.944025 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.943993 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tlnnt\"/\"openshift-service-ca.crt\"" Oct 30 22:37:33.944199 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.944072 5914 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tlnnt\"/\"default-dockercfg-w9mvk\"" Oct 30 22:37:33.944199 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.944085 5914 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tlnnt\"/\"kube-root-ca.crt\"" Oct 30 22:37:33.945637 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.945618 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tlnnt/must-gather-f9ddm"] Oct 30 22:37:33.991247 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.991218 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9fa4eaba-2591-4197-819d-53e72e0eda00-must-gather-output\") pod \"must-gather-f9ddm\" (UID: \"9fa4eaba-2591-4197-819d-53e72e0eda00\") " pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:33.991432 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:33.991271 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/9fa4eaba-2591-4197-819d-53e72e0eda00-kube-api-access-k46rh\") pod \"must-gather-f9ddm\" (UID: \"9fa4eaba-2591-4197-819d-53e72e0eda00\") " pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:34.092520 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.092436 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/9fa4eaba-2591-4197-819d-53e72e0eda00-kube-api-access-k46rh\") pod \"must-gather-f9ddm\" (UID: \"9fa4eaba-2591-4197-819d-53e72e0eda00\") " pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:34.092708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.092537 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9fa4eaba-2591-4197-819d-53e72e0eda00-must-gather-output\") pod \"must-gather-f9ddm\" (UID: \"9fa4eaba-2591-4197-819d-53e72e0eda00\") " pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:34.092915 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.092858 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9fa4eaba-2591-4197-819d-53e72e0eda00-must-gather-output\") pod \"must-gather-f9ddm\" (UID: \"9fa4eaba-2591-4197-819d-53e72e0eda00\") " pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:34.149315 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.149279 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46rh\" (UniqueName: \"kubernetes.io/projected/9fa4eaba-2591-4197-819d-53e72e0eda00-kube-api-access-k46rh\") pod \"must-gather-f9ddm\" (UID: \"9fa4eaba-2591-4197-819d-53e72e0eda00\") " pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:34.164253 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.164226 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tlnnt/must-gather-f9ddm" Oct 30 22:37:34.533746 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.533719 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tlnnt/must-gather-f9ddm"] Oct 30 22:37:34.536120 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:37:34.536094 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fa4eaba_2591_4197_819d_53e72e0eda00.slice/crio-48f7378337ad36f1ff8b3de89eb8152fd511baf23dfc7818d1be75f21c6a357a WatchSource:0}: Error finding container 48f7378337ad36f1ff8b3de89eb8152fd511baf23dfc7818d1be75f21c6a357a: Status 404 returned error can't find the container with id 48f7378337ad36f1ff8b3de89eb8152fd511baf23dfc7818d1be75f21c6a357a Oct 30 22:37:34.539265 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.539243 5914 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:37:34.788178 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:34.788088 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/must-gather-f9ddm" event={"ID":"9fa4eaba-2591-4197-819d-53e72e0eda00","Type":"ContainerStarted","Data":"48f7378337ad36f1ff8b3de89eb8152fd511baf23dfc7818d1be75f21c6a357a"} Oct 30 22:37:36.798545 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:36.798505 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/must-gather-f9ddm" event={"ID":"9fa4eaba-2591-4197-819d-53e72e0eda00","Type":"ContainerStarted","Data":"8e0daacb8d95f1e9e436485883d327893eee2e9e64c272b7de267d54d185fb12"} Oct 30 22:37:36.799003 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:36.798559 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/must-gather-f9ddm" event={"ID":"9fa4eaba-2591-4197-819d-53e72e0eda00","Type":"ContainerStarted","Data":"0aa397bd4c1b74d0eff4bb10aef9b206ab08748782d4861707f4f3936db96177"} Oct 30 22:37:36.912812 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:36.912754 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tlnnt/must-gather-f9ddm" podStartSLOduration=2.675711207 podStartE2EDuration="3.912738697s" totalImagesPullingTime="1.23702749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:36.911741132 +0000 UTC m=+4808.813605211" watchObservedRunningTime="2026-10-30 22:37:36.912738697 +0000 UTC m=+4808.814602762" Oct 30 22:37:42.344946 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:37:42.344908 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:37:43.241264 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:43.241224 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5vztw_21dcf34f-5308-47fd-a7a2-5a277aa95daf/global-pull-secret-syncer/0.log" Oct 30 22:37:46.245702 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:46.245672 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9q452_0b8f39f3-c5f2-4cde-8a5d-05cd13829521/konnectivity-agent/0.log" Oct 30 22:37:48.059907 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:37:48.059836 5914 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-119.us-west-1.compute.internal_d6458e4c6fa120c378f995c71dc796f0/haproxy/0.log" Oct 30 22:38:12.426201 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:38:12.426142 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor" Oct 30 22:38:31.147851 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.147816 5914 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n"] Oct 30 22:38:31.153708 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.153682 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.241444 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.241412 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n"] Oct 30 22:38:31.245449 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.245418 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-proc\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.245623 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.245550 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cs2\" (UniqueName: \"kubernetes.io/projected/289087c2-600b-4dbf-8970-f5a86fd5585c-kube-api-access-w6cs2\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.245623 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.245599 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-podres\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.245750 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.245685 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-sys\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.245750 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.245729 5914 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-lib-modules\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347179 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347145 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-podres\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347218 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-sys\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347247 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-lib-modules\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347262 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-podres\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347321 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-sys\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347322 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-proc\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347352 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347350 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-proc\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347563 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347355 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/289087c2-600b-4dbf-8970-f5a86fd5585c-lib-modules\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.347563 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.347403 5914 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cs2\" (UniqueName: \"kubernetes.io/projected/289087c2-600b-4dbf-8970-f5a86fd5585c-kube-api-access-w6cs2\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.409955 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.405342 5914 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cs2\" (UniqueName: \"kubernetes.io/projected/289087c2-600b-4dbf-8970-f5a86fd5585c-kube-api-access-w6cs2\") pod \"perf-node-gather-daemonset-sxk7n\" (UID: \"289087c2-600b-4dbf-8970-f5a86fd5585c\") " pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.468599 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.468567 5914 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:31.840132 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:31.840108 5914 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n"] Oct 30 22:38:31.842925 ip-10-0-142-119 kubenswrapper[5914]: W1030 22:38:31.842893 5914 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod289087c2_600b_4dbf_8970_f5a86fd5585c.slice/crio-9d1495cb1108b0fcac822970be1b39a9be35978820e12d3e259ffcef82407d33 WatchSource:0}: Error finding container 9d1495cb1108b0fcac822970be1b39a9be35978820e12d3e259ffcef82407d33: Status 404 returned error can't find the container with id 9d1495cb1108b0fcac822970be1b39a9be35978820e12d3e259ffcef82407d33 Oct 30 22:38:32.095317 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:32.095218 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" event={"ID":"289087c2-600b-4dbf-8970-f5a86fd5585c","Type":"ContainerStarted","Data":"a68575fec5db345fee195d52c0fa910a2bb49f238430b06a891aec06c09876a1"} Oct 30 22:38:32.095317 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:32.095285 5914 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" event={"ID":"289087c2-600b-4dbf-8970-f5a86fd5585c","Type":"ContainerStarted","Data":"9d1495cb1108b0fcac822970be1b39a9be35978820e12d3e259ffcef82407d33"} Oct 30 22:38:32.095551 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:32.095328 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:32.210419 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:32.210359 5914 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" podStartSLOduration=1.210345169 podStartE2EDuration="1.210345169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:32.209624179 +0000 UTC m=+4864.111488245" watchObservedRunningTime="2026-10-30 22:38:32.210345169 +0000 UTC m=+4864.112209288" Oct 30 22:38:38.118514 ip-10-0-142-119 kubenswrapper[5914]: I1030 22:38:38.118413 5914 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tlnnt/perf-node-gather-daemonset-sxk7n" Oct 30 22:38:42.345531 ip-10-0-142-119 kubenswrapper[5914]: E1030 22:38:42.345493 5914 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-7bckf" containerName="extractor"